McMahon - ECMWF Mix Run 12 z Missing - Last was 2023-10-12-06 z

Software WxSim
1

McMahon - Last Good ECMWF Mix Run was ( 2023-10-12-06 z ) and Last Should Be ( 2023-10-12-12 z ).

It is possible that the Data is running Late.

Note: this is an automated Post, check data to ensure it is correct!

Kindest Regards,

Tony

2

I was in St Helens at the time this occurred (only just got home now), so cannot validate, maybe Stuart @broadstairs can confirm if it was late.
Kindest Regards,
Tony

3

I just checked and the ECMWF data for 12th 12z data showed up by 23:00 BST.

The 18z for the 12th arrived this morning by 03:30 BST on 13th and the 13th 00z arrived by 09:30 BST

It does seem that the ECMWF data is arriving somewhat later than what seemed normal earlier in the month.

Tony Iā€™ll try to make a page you can view online to see my records of timings for Chrisā€™ data that I keep on a rolling 7 day basis. It wont be linked to my website directly so when itā€™s done Iā€™ll send you a link.

Stuart

1 Like
4

Iā€™m just taking a quick look at this. Unfortunately I havenā€™t yet modified the ECMWF script to log start/end times so I donā€™t have more detailed info as I do for GFS.

From what I can see of data remnants and files which havenā€™t been superceded yet the ECMWF data appears to be arriving later than it used to. When I wrote the ECMWF data processing scripts I made notes of when I expected the data to become available. Hereā€™s a comparison of what I originally documented and when the data appears to be downloaded now (all times UTC).

Run Hour Expected Ready Time Current Ready Time
00Z 06:27 07:51
06Z 12:12 13:16
12Z 18:27 19:51
18Z 00:12 01:15

So it looks like the data isnā€™t ready for me to download for 1-1.5 hours later than it used to be. The 00/12Z downloads have more data which is why they always took longer and thatā€™s possibly why they are even more delayed now.

Note: these are download times. My scripts still have a lot of processing to do post-download. For example, the latest (13 Oct 00Z) data was downloaded by 07:51 but my processing didnā€™t finish until 08:18. The latter time is when itā€™s available for you to access it.

ECMWF is a bit messier to investigate than GFS because the data availability check and downloading is ā€˜hiddenā€™ by some ECMWF provided Python scripts. So I need to pull the scripts apart to find the underlying data storage location(s) to check what time the data is timestamped. I just want to check when the data files really appear compared to when the scripts say theyā€™re ready. Iā€™m sure the scripts are working OK, but I just want to try to figure out where the problem lies. However, I know from previous investigation into the scripts that there are multiple locations to download the files from and Iā€™m wondering whether the one Iā€™m using is now delayed and switching to another source might be better.

I have a busy few days ahead and some other stuff that my coding head is full of right now (OpenGrads with GRIB2 data) so Iā€™m not promising a quick fix, if one is even possible. If the data is just being provided later than it used to be then there isnā€™t anything I can do about that.

5

Hi Stuart, that would be brilliant, Greatly appreciated.
Kindest Regards,
Tony

6

Hi Chris,
Sorry for highlighting this issue, hopefully an easy resolve can be found.
In the mean time do you want me to leave the Auto Api Posts checks running or pause them for say ECMWF Mix only.
Kindest Regards,
Tony

7

Leave them all running. Others might ask why the data is missing/delayed so the updates may help them.

I did wonder whether it was my GFS and ECMWF scripts clashing, but I doubt thatā€™s the case. There may be occasions when they run in parallel but that shouldnā€™t be 4 times every day. I also donā€™t think itā€™s my backups eating up bandwidthā€¦that would at most only affect one of the 4 runs each day. So Iā€™m thinking itā€™s just the data thatā€™s being delivered later than it used to be. ECMWF donā€™t seem to have put the same effort into providing seamless and quick access to data which is what the GFS data service is these days, so it wouldnā€™t surprise me if it was just a case of providing the cheapest service whilst still saying itā€™s made available!

8

Cheers Chris yes I agree, will do.
Totally agree your latter point is a high probability (cheapest service).
Kindest regards,
Tony

9

I think that we might need to re-assess the ECMWF availability times. The official info from ECMWF is that ā€œThe (open) data are released 1 hour after the real-time dissemination scheduleā€. Iā€™ve created a table to show this data below (allowing for 30 minutes of processing in the W-W availabilty once Iā€™ve downloaded all the data):

Run Hour Real Time Avail Open Data Avail Open Data Actual Avail Expected W-W.com Avail
00Z 06:12-06:27 07:12-07:27 07:51 08:21
06Z 11:45-12:12 12:45-13:12 13:16 13:46
12Z 18:22-18:27 19:22-19:27 19:51 20:21
18Z 23:45-00:12 00:45-01:12 01:15 01:45

The Actual/Expected W-W availability times are from a single day of download timings so thereā€™s likely to be some variability in those numbers. However, thatā€™s suggesting that 00/12Z arenā€™t becoming available until getting on for 8.5 hours after the ā€˜run start timeā€™. This is a little late (30 minutes or so) beyond time we could expect if the ECMWF data was available on time. For 06/18Z itā€™s about 7.75 hours after run start time. This is just beyond the latest time we could expect for data availability.

Tony - how do these times agree with what youā€™re expecting to see?

Note: This is currently only a theoretical analysis without examining the scripts and processing. I will get to that before much longer, but Iā€™m not sure Iā€™ll be able to shave much time off in my scripts if the data is really arriving 30 minutes later than predicted.

10

Hi Chris,
Great thanks for this.
OK as I have already pushed my times back 1 Hour (and we are currently in DST), I have converted my current Local (AEDT) Run Times to UTC called (Tony UTC/in DST Hour).
my future Local (AEST) Run Times to UTC called (Tony UTC/in ST Hour).

Run Hour Real Time Avail Open Data Avail Open Data Actual Avail Expected W-Wcom Avail Tony UTC/in DST Hour Tony UTC/in ST Hour
00Z 06:12-06:27 07:12-07:27 7:51 8:21 9:06 10:06
06Z 11:45-12:12 12:45-13:12 13:16 13:46 15:06 16:06
12Z 18:22-18:27 19:22-19:27 19:51 20:21 21:06 22:06
18Z 23:45-00:12 00:45-01:12 1:15 1:45 3:06 4:06

Will be running Not too Close to the wind now in DST, though OK if not a tad late in ST, though realy donā€™t want to change run times every DST/ST change itā€™s a pain, so planning on leaving them as is for now.
Hope that all makes sense.
Kindest Regards,
Tony