McMahon - Last Good ECMWF Mix Run was ( 2023-11-11-18 z ) and Last Should Be ( 2023-11-12-00 z ).
It is possible that the Data is running Late.
Note: this is an automated Post, check data to ensure it is correct!
Kindest Regards,
Tony
The 00z data is marked as being available at 09:15 (UTC).
1 Like
Yeah, was later than normal.
So auto flagged as not ready when my Run occurred which is possibly an hour later than others.
I assume there’s quite a delay between you checking if the data is available and when you send the message? Your message was posted about 7-8 minutes after the data was available (based on the timestamp on the report file which is produced as the ver last action of the processing run).
WxSimate downloads at :06 WxSim-Lite runs at :08 and WxSim runs at :10 (Then after complete uploads to weather site normally before :15 then crons check at :20 and :25 and :26 and :27 in case the upload was late.
Which step creates the forum post and at what point in the process? Your WxSim processing started before data was available, but the post to say the data was missing happened about 7-8 minutes after the data became available. So does that mean your WxSim processing took about 17 minutes to complete and then the post was made about missing data? If this is correct, then I wonder if it’s worth changing the post to say that data was unavailable at “HH:MM”, e.g. at the start of WxSim processing run.
I’m not complaining! Just trying to understand the process and what (if anything) I can do to improve things.
Hi Chris,
Sorry late reply, crashed early.
Yeah that’s fine I undersatnd.
Hopefully this helps, All GFS/ECMWF (source files) are imported via WxSimate which starts import (for me at hh:06 more detail below).
I can try an look at adding a time, though may be fun and games, will see what I can do.
| UTC | AEDT | ||||||
|---|---|---|---|---|---|---|---|
| Runs | Tony’s Run Times | GFS | ECMWF | ||||
| 00z | 12/11/2023 06:06 | 12/11/2023 17:06 | ĂĽ | 00z | ĂĽ | 18z | |
| 00z | 12/11/2023 09:06 | 12/11/2023 20:06 | ĂĽ | 00z | ĂĽ | 00z | WxSimate Starts Import of all Data : 09:06 UTC |
| 06z | 12/11/2023 12:06 | 12/11/2023 23:06 | ĂĽ | 06z | ĂĽ | 00z | |
| 06z | 12/11/2023 15:06 | 13/11/2023 02:06 | ĂĽ | 06z | ĂĽ | 06z | WxSimate Starts Import of all Data : 15:06 UTC |
| 12z | 12/11/2023 18:06 | 13/11/2023 05:06 | ĂĽ | 12z | ĂĽ | 06z | |
| 12z | 12/11/2023 21:06 | 13/11/2023 08:06 | ĂĽ | 12z | ĂĽ | 12z | WxSimate Starts Import of all Data : 21:06 UTC |
| 18z | 13/11/2023 00:06 | 13/11/2023 11:06 | ĂĽ | 18z | ĂĽ | 12z | |
| 18z | 13/11/2023 03:06 | 13/11/2023 14:06 | ĂĽ | 18z | ĂĽ | 18z | WxSimate Starts Import of all Data : 03:06 UTC |
Also need to confess that ever since Mix started I had failed to set it up correctly, which in essence resulted in no mixing all that time. It was only when I found the issue with DST not working last month that I realised. Since then I have tested and found that my Forecast Accuracy acully became worse with ECMWF mixed… Maybe it’s a Southern Hemisphere thing I don’t now. However I realy wanted to just go back to full 100% gfs, Though I had spent so much time on all the scripts and reporting when one run etc was missing/late that I felt compelled to keep it all running for that reason. The only way I can do that is how I am know reduced ECMWF to 10% so it’s minimal but actually importing the data and including it in the mix output files which is where I do the real final checks to ensure the run/data was actually used in the forecast.
Hopefully all that makes some kinda sense 
Kindest Regards,
Tony
Hi Chris,
Firstly my apologies, I have had so many personal issues of late that additional work on this has sadly had a backseat.
I have a lot going on right know though will find time in the next week or so to create some new checks Based Purely on What times YOU determine they should be available.
That way it is not what my initial Long Standing Checks has been purely based on (if my forecast had correct data included).
I have removed my ECMWF Mix on my site, so for now any Automated Posts will be if ?
1:Bohler-Lite Data is Missing.
If I select Bohler for GFS/Data/Adv.
2: Bohler GFS Data/Adv is missing.
else: If I select McMahon for GFS/Data/Adv.
3: McMahon GFS Data/Adv is missing.
Over the next week or more, I will endeavour to create some new (checks / posts) based on (If you provide YOUR proposed Availability Times). I will make that so I can easily adjust the Times if you advise a change.
The Checks would be Simply. Is McMahon (ECMWF or GFS) available, based on the content of your 2 *_report.txt files.
I think at least that way it is relevant to all users, rather than just to (my if the correct Run was included in My personal actual forecast).
Below is all that will be displayed on my site for now (oh I have selected McMahon for (GFS/Adv).
Note: I should not probably say it, though wonder if Late Arrival of Data by the big Boys is a ploy to say (Hey you can get it earlier if you pay) ?
Who knows matey.
Kindest Regards,
Tony
Hi Tony
Firstly, there’s no need to rush to make changes if you’re busy. This is just a hobby for all of us, and real-life always takes precedence.
Secondly, you don’t need to make huge changes to your scripts. I suggested using your process for reporting because you already had most of what was needed in place. I will eventually find time to build my own scripts to better reflect the timings, availability and stats related to the GFS and ECMWF data. Like you, I’ve also been busy with other things so this work hasn’t risen to the top of my pile yet.
I will suggest a minor change that might be easier for you to implement rather than creating a whole new set of scripts. I now better understand your reporting process and how it’s caused me confusion. If I’m available (which is a lot more these days) then when I see your automated posts about late data I start investigating to see if there’s a problem. In quite a few cases recently I’ve found that your forum post is made 5-10 minutes after the data was actually available. I now understand that’s because you try to download data, then process whatever data you have in WxSim and at the end of that process report that data was missing. It can take 10-15 minutes between you checking that the data was missing and reporting that, but in that time the data becomes available.
The problem is that we’re using the data for two different things. You are using it to note that your WxSim processing didn’t have all the data it needed, but don’t otherwise take any short-term action. I’m using the posts as triggers to immediately check (if I’m available) whether there are problems with data processing that I’m not aware of. I do have some checks of my own, but they tend to trigger later because of the vagaries of data availability times from GFS and ECMWF.
So perhaps as an alternative to a major script re-write could you just add a quick check of the report file that is produced by each successful run immediately before making the forum post? Perhaps the forum post could be changed slightly to something along the lines of “ECMWF data wasn’t available when requested, but is now available”?
Hi Chris,
Thanks greatly appreciated.
Yes I agree with all your comments, will do.
I just noticed that both McMahon (GFS/ECMWF) 12z Runs where an hour earlier today than yesterday (using Stuarts Log).
Kindest Regards,
Tony