Chris I’m just wondering why the ECMWF data is now arriving somewhat later than prior to the DST start here in the UK? I was running my forecasts at 02:00 08:00 14:00 and 20:00 and was getting both GFS and ECMWF for the same timing pretty much consistently but now ECMWF seems to be about an hour to an hour and a half later. GFS has not changed its timing!
Any ideas?
Stuart
The stats don’t seem to be showing data being later than usual.
The 00/12z ECMWF data started running about 15 minutes later than the long term (3 month) average but that started about 6-8 weeks ago (so long before the DST change) after ECMWF changed to providing 0.25 degree data rather than the previous 0.4 degree data. The 00/12z runs have more data (144 hours instead of just 90 hours) so I took that to be the reason why those runs were taking longer. Actually the data availability time from ECMWF was what slipped…my own processing of 0.25 degree data is still pretty much the same as it was for 0.4 degree.
Other than that the data seems to be arriving and processing pretty much to time for the last month and I’m certainly not seeing anything like the 60-90 minute delays you’re seeing (based on the stats).
I record the availability of GFS and ECMWF data from your site and 15 minute intervals. You can see my data for the last 7 days at https://www.stella-maris.org.uk/CumulusMX/GFSdata/gfs.phtml and it shows that for example that yesterday the 18Z ECMWF was only available by 02:45 it used to be 01:45 but I know that these timings could mean the data appeared up to 14 minutes earlier. Other timings show the 09Z was not there until about 09:45 but would have previously been there in time for a forecast at 08:00 probably showing as available by 07:45. Obviously I am trying to time my forecasts to coincide with the best data availability which as I said used to be 02 08 14 and 20, which is no longer true.
So I am seeing this ECMWF data appearing later than it was. I realise it is what it is but the change in timings does seem quite large. I don’t think for a minute it is your servers but ECMWF themselves.
Stuart
That’s the GFS download/processing which went on to finish 4 minutes earlier than the long term average.
The server processing the GSF and ECMWF data is configured to use UTC so that it’s not affected by DST changes. I’ve just confirmed that setting…
root@hv1 /home/dash/public_html # cat /etc/timezone
Etc/UTC
The timings for WxSimate data processing performance use the value of time() which provides a UTC based timestamp, e.g.
my $now = time();
$run_history_h = $dbh->prepare(
"INSERT IGNORE INTO mmap_run_history
(run_id, run_hour, run_end_time, run_model)
VALUES ('$this_run_id', '$cycle', '$now', 'E2')"
);
So my database of timings is fully UTC based.
My stats dashboard outputs the times using PHP gmdate() which outputs a GMT (effectively UTC) time, e.g.
if ($e2_end_time > 0)
{
echo gmdate("Y-m-d H:i \U\T\C", $e2_end_time);
}
So the server, stats database and dashboard are all using UTC. As configured, the server has no comprehension of DST so it can’t really be responsible for a change at the time DST took place in the UK. In any case, the server is based in Finland, so as built (before my configuration) it would currently be operating on UTC+3, so if I’d missed configuring something then the time would be significantly wrong!
I’ve also just checked the ‘raw’ ECMWF data, i.e. my download source, to see when the data became available for me to download for the last 4 complete runs (06z for today was running as I typed)…
| Run | ECMWF | Data Ready | Complete | Stuart |
|---|---|---|---|---|
| 2024040906 | 13:12 | 13:15 | 13:36 | 14:45 |
| 2024040912 | 19:55 | 19:55 | 20:21 | 21:30 |
| 2024040918 | 01:12 | 01:15 | 01:37 | 02:45 |
| 2024041000 | 07:55 | 07:55 | 08:22 | 09:30 |
The ‘ECMWF’ column shows the time (UTC) of the last (90h or 144h) file from each run on the ECMWF server. ‘Data Ready’ is the time my script recorded as seeing the last (90/144h) file become available. My script checks every 5 minutes to see if data is ready so it will show the next 5 minute step after the ECMWF time. ‘Complete’ column shows the time (UTC) that my processing completed - this is the time when the data becomes available for download by WxSimate. ‘Stuart’ shows the times you’re recording from your web page. You would normally show the 15 minute interval following the ‘Complete’ column, but in each case you’re showing data available to WxSimate an hour later than my data shows.
If you look at my dashboard it’s showing little variability in timings between the 2 week and 1 month averages, and relatively little between the 2 week/1 month and the 3 month averages (apart from the difference noted previously about the change to 0.25 degree data. It definitely doesn’t show a difference between timings pre-UK DST and post-UK DST, which would be significantly affecting the 2 week average stats by now.
Can you please confirm that all your dates are definitely being stored and displayed as UTC values. If they are then I’m not able to account for what’s causing the apparent time warp between us!!
All I do is to interrogate your pages for which data is available either gfsn_report.txt or ecmwf_report.txt and as my system runs on BST in summer that is the current UK time it records not UTC at present which explains the confusion. So basically in summer I need to run each forecast about 1 hour later and switch back in winter.
I just wish we did not switch times, I’d much prefer GMT+1 for all year round!
Stuart
I’m glad there’s a good explanation of why you saw this issue!
I agree…DST is a bit of history that I don’t think we really need these days. Other countries manage perfectly well without it…and some countries take it even further, e.g. China, a country spans 5 times zones, yet the entire country runs on a single time zone without DST.
I’m not sure the powers-that-be will see the sense in geting rid of it during my lifetime though 