McMahon - ECMWF Mix Run 12 z Missing - Last was 2023-10-12-06 z

McMahon - Last Good ECMWF Mix Run was ( 2023-10-12-06 z ) and Last Should Be ( 2023-10-12-12 z ).

It is possible that the Data is running Late.

Note: this is an automated Post, check data to ensure it is correct!

Kindest Regards,

Tony

I was in St Helens at the time this occurred (only just got home now), so cannot validate, maybe Stuart @broadstairs can confirm if it was late.
Kindest Regards,
Tony

I just checked and the ECMWF data for 12th 12z data showed up by 23:00 BST.

The 18z for the 12th arrived this morning by 03:30 BST on 13th and the 13th 00z arrived by 09:30 BST

It does seem that the ECMWF data is arriving somewhat later than what seemed normal earlier in the month.

Tony I’ll try to make a page you can view online to see my records of timings for Chris’ data that I keep on a rolling 7 day basis. It wont be linked to my website directly so when it’s done I’ll send you a link.

Stuart

1 Like

I’m just taking a quick look at this. Unfortunately I haven’t yet modified the ECMWF script to log start/end times so I don’t have more detailed info as I do for GFS.

From what I can see of data remnants and files which haven’t been superceded yet the ECMWF data appears to be arriving later than it used to. When I wrote the ECMWF data processing scripts I made notes of when I expected the data to become available. Here’s a comparison of what I originally documented and when the data appears to be downloaded now (all times UTC).

Run Hour Expected Ready Time Current Ready Time
00Z 06:27 07:51
06Z 12:12 13:16
12Z 18:27 19:51
18Z 00:12 01:15

So it looks like the data isn’t ready for me to download for 1-1.5 hours later than it used to be. The 00/12Z downloads have more data which is why they always took longer and that’s possibly why they are even more delayed now.

Note: these are download times. My scripts still have a lot of processing to do post-download. For example, the latest (13 Oct 00Z) data was downloaded by 07:51 but my processing didn’t finish until 08:18. The latter time is when it’s available for you to access it.

ECMWF is a bit messier to investigate than GFS because the data availability check and downloading is ‘hidden’ by some ECMWF provided Python scripts. So I need to pull the scripts apart to find the underlying data storage location(s) to check what time the data is timestamped. I just want to check when the data files really appear compared to when the scripts say they’re ready. I’m sure the scripts are working OK, but I just want to try to figure out where the problem lies. However, I know from previous investigation into the scripts that there are multiple locations to download the files from and I’m wondering whether the one I’m using is now delayed and switching to another source might be better.

I have a busy few days ahead and some other stuff that my coding head is full of right now (OpenGrads with GRIB2 data) so I’m not promising a quick fix, if one is even possible. If the data is just being provided later than it used to be then there isn’t anything I can do about that.

Hi Stuart, that would be brilliant, Greatly appreciated.
Kindest Regards,
Tony

Hi Chris,
Sorry for highlighting this issue, hopefully an easy resolve can be found.
In the mean time do you want me to leave the Auto Api Posts checks running or pause them for say ECMWF Mix only.
Kindest Regards,
Tony

Leave them all running. Others might ask why the data is missing/delayed so the updates may help them.

I did wonder whether it was my GFS and ECMWF scripts clashing, but I doubt that’s the case. There may be occasions when they run in parallel but that shouldn’t be 4 times every day. I also don’t think it’s my backups eating up bandwidth…that would at most only affect one of the 4 runs each day. So I’m thinking it’s just the data that’s being delivered later than it used to be. ECMWF don’t seem to have put the same effort into providing seamless and quick access to data which is what the GFS data service is these days, so it wouldn’t surprise me if it was just a case of providing the cheapest service whilst still saying it’s made available!

Cheers Chris yes I agree, will do.
Totally agree your latter point is a high probability (cheapest service).
Kindest regards,
Tony

I think that we might need to re-assess the ECMWF availability times. The official info from ECMWF is that “The (open) data are released 1 hour after the real-time dissemination schedule”. I’ve created a table to show this data below (allowing for 30 minutes of processing in the W-W availabilty once I’ve downloaded all the data):

Run Hour Real Time Avail Open Data Avail Open Data Actual Avail Expected W-W.com Avail
00Z 06:12-06:27 07:12-07:27 07:51 08:21
06Z 11:45-12:12 12:45-13:12 13:16 13:46
12Z 18:22-18:27 19:22-19:27 19:51 20:21
18Z 23:45-00:12 00:45-01:12 01:15 01:45

The Actual/Expected W-W availability times are from a single day of download timings so there’s likely to be some variability in those numbers. However, that’s suggesting that 00/12Z aren’t becoming available until getting on for 8.5 hours after the ‘run start time’. This is a little late (30 minutes or so) beyond time we could expect if the ECMWF data was available on time. For 06/18Z it’s about 7.75 hours after run start time. This is just beyond the latest time we could expect for data availability.

Tony - how do these times agree with what you’re expecting to see?

Note: This is currently only a theoretical analysis without examining the scripts and processing. I will get to that before much longer, but I’m not sure I’ll be able to shave much time off in my scripts if the data is really arriving 30 minutes later than predicted.

Hi Chris,
Great thanks for this.
OK as I have already pushed my times back 1 Hour (and we are currently in DST), I have converted my current Local (AEDT) Run Times to UTC called (Tony UTC/in DST Hour).
my future Local (AEST) Run Times to UTC called (Tony UTC/in ST Hour).

Run Hour Real Time Avail Open Data Avail Open Data Actual Avail Expected W-Wcom Avail Tony UTC/in DST Hour Tony UTC/in ST Hour
00Z 06:12-06:27 07:12-07:27 7:51 8:21 9:06 10:06
06Z 11:45-12:12 12:45-13:12 13:16 13:46 15:06 16:06
12Z 18:22-18:27 19:22-19:27 19:51 20:21 21:06 22:06
18Z 23:45-00:12 00:45-01:12 1:15 1:45 3:06 4:06

Will be running Not too Close to the wind now in DST, though OK if not a tad late in ST, though realy don’t want to change run times every DST/ST change it’s a pain, so planning on leaving them as is for now.
Hope that all makes sense.
Kindest Regards,
Tony