Hi Everybody,
Since late September, I’ve been running three parallel versions of WXSIM: 0, 60, and 100% European model (the rest GFS). To keep things clean and simple, I turned off WXSIM-Lite usage (as it puts more GFS in there), and also FOUS and MOS (which are U.S. only). These omissions probably degrade my accuracy some, but allow for a more clear comparison between GFS and ECMWF, and a better idea of the ideal mix.
I don’t have definitive results yet, because periodically one of these versions has failed (lately because of the relative humidity problem in ECMWF, which Chris has now protected against. Sometime soon, I’ll go through and produce a clean, totally in-common set of forecasts (over 400 of them) for a proper comparison.
However, here’s what I’m seeing rigth now. Mean absolute error (through day4) for pure GFS is 3.00 F, 60:40 ECMWF:GFS shows 2.79, and pure ECMWF is 2.68! The suggestion there would be to go 100% ECMWF.
BUT, that’s not a complete apples to apples comparison, and I have a hunch the GFS HAS to be worth something in the mix. Pretty soon, I’ll turn WXSIM-Lite back on, and see how that plays in. It will likely improve accuracy further, but might confuse the answer to the question of best percentage of ECMWF. Right now, I’m thinking at least 60% ECMWF, but with GFS-based WXSIM-Lite in effect … though maybe at a slightly lower percentage (50? Most people use 60 or 65, I think).
Do any of you have any observations of experience to suggest the best mix?
Thanks!
Tom
Hi Tom,
1: I apologise for not (at present) being able to provide you with good analytic data as I have been tied up with visiting family and friends.
2: I have noticed just recently that many Runs have been malformed, though I have not been able to email / send info regards same, due to the above.
3: I have been running with GFS/20% and ECMWF/80% and not stopped using WXSIM-Lite!!
I should Add here that I still run re-forecasts every 3 hours, so that means ECMWF is not aligned in the Mix every 3/6 Hours.
4: I am off to hospital this week and will not respond for about a week, though once I return home will try to do some analysis if I can
5: In the mean time here is my live Comparisons to Forecast vs Actual post the recent Run Availability issues I have seen, prior they where a tad ordinary.
Note: More links below image 8)
Image :
6: Live Link : https://beaumaris-weather.com/charts_Combined_Forecast_Actual.php
7: Link to Last 7 Days Temperature Differential (WxSim Forecast vs Actual). https://beaumaris-weather.com/wxsimcsvmysql/realtime_graphs_wxsim_Act_WX_Delta_include.php
8: Errdate and Errdep achieve images reside here if you wish to view them. (I suppose they tell some of the story) 
https://beaumaris-weather.com/data/wxsimerrimages/
Kindest Regards,
Hi Tom,
It seems maybe no one is either using the (GFS/ECMWF) Mix or has not done any analysis on their usage of it.
I personally found that due to the offset arrival of ECMWF Data that it did not prove beneficial to me.
I have always used Autolearn (since the early days of beta testing it) and WxSimate.
I prefer to run WxSim every 3 Hours as I believe that local Data injected into WxSim along with it’s Learning ability (using overnight Autolearn determined factors) results in the best local forecast overall. Whereas say the BoM only reforecast on much greater periods.
I would love to see a Global Poll which asks the following regards (WxSim (GFS/ ECMWF) Usage ) and if used, at what frequencies and at what (GFS/ ECMWF / percentage %).
Chris is it possible for this Poll to be done.
Kindest Regards,
Tony
I’ll try to create something later, although it won’t realistically be possibly to catch all possible combinations.
For example, I can have options for 100% GFS / 0% ECMWF, 90/10, 80/20, etc, but of someone uses 72/28 that would need 101 options to capture all possible values. I think it will need to ask what’s closest to the split.
It also wouldn’t be possible to capture somebody doing what Tom does, i.e. multiple parallel runs with different parameters.
I’ll experiment when I’m on my laptop later on.
Oh Brilliant thanks Chris.
I suppose it would be interesting to know if users have done any analysis of the before / after usage of the mix, has it improved the quality of their forecast or the opposite or no change etc…
Sorry should have added this earlier.
Kindest Regards,
Tony
The best way to test it is in parallel, running exactly the same forecasts, with and without the ECMWF data. I’ll have to go back and look at what I found (from several hundred forecast, spanning something like 7 months), but at least for my location (Atlanta) the ECMWF pretty clearly helped. In fact, the 100% ECMWF was better than any of the mixes. I suspect that may be a bit of a fluke, as I’m almost willing to bet that at least some GFS in there is beneficial.
Once I was pretty sure ECMWF was helping, I changed my parallel runs so that all of them are 70:30 ECMWF:GFS, but with different levels of WXSIM-Lite usage. I haven’t analyzed that let, but will do so soon.
To do parallel runs with different settings, I just set up three different folders. I have them running with the exact same data, staggered by about 10 minutes. I had a really good streak lately, with (I think) all three instances running with no error messages 5 times a day, for two solid months. Then one of them had a glitch a few days ago and I had to restart it.
This provides an apples-to-apples comparison, which is more of a fair comparison that before and after a change of settings (even though that can be quite useful, too). Anyway, I would be curious to hear feedback, from anyone who’s done a careful comparison. I suspect results will vary some with geographic location. The “tricky” places will probably be coastlines or right next to mountain ranges, where some difference in the two models’ grid point may favor one over the other.
Thanks!
Tom