McMahon - Last Good ECMWF Mix Run was ( 2023-08-29-06 z ) and Last Should Be ( 2023-08-29-12 z ).
It is possible that the Data is running Late.
Note: this is an automated Post, check data to ensure it is correct!
Kindest Regards,
Tony
I just checked my script and Chris’ ECMWF appeared by 21:15 as it has been now for most days. My script logs the data availability every 15 minutes so it will have been appeared between 21:00 and 21:15 (timings are BST not GMT(UTC) at present).
Since running this script for a couple of weeks now I’ve been starting my data downloads and forecasts at 06,10,15 and 22 on the hour as that seems to coincide with both GFS and ECMWF being the same time periods.
Stuart
Hi Stuart, @Tom_Ehrensperger @samuelbohler
Yes it was available, I just woke up and saw this.
All of a sudden WxSimate is hanging on downloading Bohler Data/Adv which has never occurred before.
This results in freezing WxSimate and not downloading any further EUROMix Data etc.
I have tested it 6 times just now and fails every time. But does not fail for McMahon Data/Adv.
I can easily download/view the Bohler Data/Adv on Tom/Sams server and download it manually, but not with WxSimate. Totally weird why all of a sudden this has started to occur.
UPDATE : “I can easily download/view the Bohler Data/Adv on Tom/Sams server” Correction, Data is OK downloading with WxSimate and viewing via Browser, however Adv continually shows loading. ?
Note : I did raise the following ages ago with Tom/Sam/Chris, being the difference between my downloads of McMahon vs Bohler Adv files.
I finally after ages waiting obtained the latest Bohler (Adv) and have attached it along with McMahon (Adv) txt files, they seem to be incomparable.! Same as what I found back then when I raised it, As you mentioned recently Chris your scripts where built for other project prior to WxSim and Sam built his for WxSim. I have always used Bohler primarily because I use WxSim-Lite so made sense to use the same source. But when the current issues with Bohler arise I have to use McMahon beit better or worse.
Downloads.zip (633.3 KB)
Now returns error after letting run for about 15 mins.
Hi Stuart,
Just curious how you are doing the “My script logs the data availability every 15 minutes” so are you hitting the servers every 15 mins ?
My script just reads two report files which Chris’ scripts update on completion of each run to show the currently available data.
https://wxd.weather-watch.com/gfsn_report.txt
https://wxd.weather-watch.com/ecmwf_report.txt
So it doesn’t actually read the data. Currently my results from this are only kept locally not uploaded anywhere as my intention was to find out optimum times to run forecasts I currently keep 7 days worth of report files.
It would be good if Sam’s scripts created the same small text files to show what has just been downloaded.
Stuart
I’m not at my laptop so I can’t look at the attachment yet on my small phone screen. Are the files what you’ve downloaded manually or are they what WxSimate has downloaded? If manually downloaded can you please send me a message with the URLs you used to download them.
Tony
From checking my web server logs I suspect you’re comparing the data by manually downloading the data using a browser because I can see both WxSimate and browser access to the GFS advection URL for your location. If I’m right then you’re picking up the wrong GFS advection data from my URL because you’re using the wrong coordinates, i.e. you’re manually downloading it for 148.27W rather than 148.27E. So it’s not surprising that my data and Sam’s data look different!
For historic reasons the coordinates used by my scripts are weird. The coordinate values are inconsistent between home data and advection data, but WxSimate knows how to use the correct values in the correct places. The URLs were never intended to be used for humans to access the data so we’ve never bothered to fix the long-standing issue because WxSimate was coded to work around it. There was a good reason for them being different way back around 2005, and I think the intention was that we’d eventually bring them back in line, but we never did. I guess we adopted an “if it ain’t broke don’t fix it” approach.
I don’t know if the same applies to Sam’s data. Possibly not, but again WxSimate is coded to use the correct coordinates to get the data it needs from Sam’s site.
For your location, longitude should be +ve for the GFS and ECMWF home location data and -ve for GFS and ECMWF advection data.
Chris out of interest I have just downloaded the GFS & ECMWF using WXSimate from both your site and Sam’s. Extracted just the unmixed data and ran a compare and there are differences in the data between the two. Part of the difference is the final column which only exists in your data but there are differences in the actual data lines as well. Some of those are only in a decimal point values where one has .1 and another .2 but there are other differences which are greater like one shows 74.8 and 88.1 in the same field on line 30.
Now I do not profess to know whether these differences are significant or not, I just put tis out as information for those who know better to explain.
McMahon-gfs-ecmwf.txt (14.6 KB)
Bohler-gfs-ecmwf.txt (14.1 KB)
Stuart
The last column is easy…it’s the Ozone data. If you look in WixSimate it says that you can optionally get the Ozone data from McMahon GFS rather than downloading it separately. This value hasn’t been added to Bohler data. I don’t know if it’s planned to do this though.
Looking at other differences…
- APCP - Precipitation varies by +/- 0.01mm due to rounding differences. Not significant.
- TCDC - Cloud cover. This is an issue that I’ve discussed with Tom before. I believe that Sam is calculating this incorrectly due to a change in the way GFS data was provided a little while ago. Tom will discuss this with Sam when Sam is less busy, but has also said that cloud cover values don’t significantly affect the forecast so he doesn’t see it as a big issue.
- WSPD - Wind speed varies by +/- 0.1m/s due to rounding differences. Not significant.
- WDIR - Wind direction varies by up to +/-5 degrees, but usually significantly less. I think this is due to differences in way Sam and I do the vector maths which has to be used to calculate the direction. I don’t think these differences are significant but Tom would give a better answer to this.
Rounding and other small differences are probably inevitable when processing large volumes of data, particularly when the code was developed independently on two very different environments (I believe Sam may be using MS products on Windows and I’m using Linux with Perl/PHP). I know Tom has compared our data on more than one occasion and hasn’t found anything to be concerned about
Chris I appreciate you explanation. I know that rounding will affect some results as I was not bothered about 0.1 differences in readings or any other minor point differences. Now I understand the other more significant one.
Thanks again for putting my mind at ease…
Stuart
I’ve just made a small change which might make no difference to the WDIR calculation or might bring my values closer to Sam’s. Having looked at my calculation it’s possible that mixing integers and floating point values in the same calculation are making things a little less exact than they could be. Adding a decimal place to some values, e.g. 180 to 180.0 may make no difference, but if it allows extra precision by all the numbers in the calculation being floating point then it could change things in a beneficial way. All values are rounded to the required level of precision after claculation so you won’t see any extra decimal places in the output.
I’ve been having trouble over the last couple of days with WXSIMATE hanging during a forecast using Bohler. This is the error message:
Last night I changed to McMahon and I’ve had good forecast runs since then.
I think that’s just a coincidence. The error you’re getting relates to RAOB data and my server (McMahon) isn’t involved in providing RAOB data.
If it happens again, try changing the setting against the RAOB line in WxSimate. Mine is set to the proxy.wxsim… option and that’s working OK.
Hi Chris/Stuart,
Apologies for late reply, I had to urgently leave the property yesterday and did not get back to very late last night.
Whatever was causing the Bohler Adv (only as Data and Lite were fine) to hang has now ceased and downloading fine.
Not heard back from Tom/Sam on this so no idea what happened maybe a corrupt file or 2 I think.
Thank you so much for the detailed info and mods to McMahon Adv (greatly appreciated) and like Stuart for the explanations.
With ref to quote above regards the Long W/E issue, I confirm that WxSimate downloads McMahon Adv correctly with the Long (=- ) rather than Long (=). However the reason I used (=) for the example yesterday was I was just using the links you had sent me ages ago which where Long(=) https://discourse2.weather-watch.com/t/re-mcmahon-ecmwf-mix-run-18-z-missing-last-was-2023-06-28-06-z/71827/5?u=beaumariswx I was totally unaware of the (=-) issue, thanks for clearing that up.
Kindest Regards,
Tony
Note : I marked this as Solution, although slightly incorrect as the main issue here for me was that Bohler Adv would not download, cause still un-known.
Oh,
I re-Ran WxSimate using McMahon and Bohler, attached both files for compare (I use DiffMerge).
Much Much better than the incorrect Manually Downloaded Incomparable ones.
I had to Manually download yesterday as WxSimate would not download from Bohler.
Just did not know about the Long(=-) for Adv for McMahon.
Thanks again for all the help, appreciated.
Bohler_2023-08-30-18_gfsadv_wxsimate.txt (1.1 MB)
McMahon_2023-08-30-18_gfsadv_wxsimate.txt (1.1 MB)
Sorry for the confusion. I’m pretty sure that I copied the GFS home location URL (+ve) and pasted it three times, just editing the script name in each case and forgot to modify the coordinates for the advection URLs.
OK I am not starting a new thread here, just tagging onto this one.
Once posted I will email Sam/Tom. @Tom_Ehrensperger , @samuelbohler
Just Auto Ran WxSim GFS 2023-08-31-00 z and my auto checks highlighted that Bohler Adv had not downloaded.
This was not entirely true, It did though not in the normal timely manner, which resulted in all the following downloads for WxSimate being late for inclusion in the forecast. The bad point here is that WxSimate does not flag an error and the normal user would not even know this has happened.
This is what happened recently for me and now again today on this run, Bohler Adv hangs even hours after it should be final.
It downloads OK again now, but not at time of normal Auto Run.
Below is output from my automated checks, Note Bohler GFS-Adv ( 18 z ) should be ( 00 z ). I get emailed when these issues occur.
Error - Bohler WxSim issue.
Last WxSim-Lite - GFS was ( 00 z )
Last Wxsimate - Bohler GFS was ( 18 z )
Kindest Regards,
Tony