Upcoming version, and test results

Let's Chat
, , ,
1

Hi Everybody,

I’m planning to produce an update to WXSIM fairly soon. I’ve made one change at this point, as per a user request: the solar panel power prediction algorithm can now accommodate sets of panels facing in up to three different directions (previously it was just one). I don’t have any other changes in mind at the moment, but would like to “open the floor” here to suggestions!

Also, for the last two months, I’ve been testing different percentage usages of WXSIM-Lite (which is derived from GFS, with bias corrections learned from “experience”) in conjunction with the relatively new ECMWF data (used at 70% versus 30% GFS, which seems to be near-optimal, based on a 7 month study I did previously). I’m finding so far that WXSIM-Lite still improves forecast accuracy, even with ECMWF included. The improvement is less dramatic than in the context of GFS alone (because ECMWF yields distinct added value, and seems to be a bit more accurate than GFS), but still significant, I think. Temperature error reduction with WXSIM-Lite seems to be optimized now somewhere around 50% weighting (rather than the 60-70% from previous studies), and is on the order of 5%. I’m still working on this, but I think 70:30 ECMWF:GFS weighting and 50% use of WXSIM-Lite may be optimum, and likely yields the best performance yet for WXSIM!

All ears here … and thanks for using WXSIM!

Tom

2

Tom,

Back in July 2021, you mentioned you were thinking of developing a tool that allowed the user to determine the best Wxsim-lite mix for his environment (https://discourse2.weather-watch.com/t/deciding-optimal-value-for-percentage-of-wxsim-lite-use/69479). Have you given any more thought to this tool? Or perhaps a tool regarding the ECMWF vs GFS mix?

Steve

3

Hi Tom,
I will add much more to this Thread over time, though initially I would like to say thank you for WXSIM (So Far).

I received your email and will respond in detail over the next week.
I agree that things will vary over time (Seasonally), however at this time I am so stoked at the quality of WXSIM/EUROMIX results (Especially looking out at the 7 + Day Forecast - Unreal)

https://beaumaris-weather.com/wxsimforecast_accuacyP7Stats.php
or
https://beaumaris-weather.com/wxsimcsvmysql/wxsim_Act_Delta_Table_allz_Date_P7Stats2_SA.php

image
image1657×2042 486 KB

I have had brilliant 7 + Day forecast in the Past, though never this Good :wink:

ECMWF Mix : started at 50% on December 27, 2022 and changed to 80% on April 30, 2023.
WXSIM-Lite Mix : set at 55 though wavers (currently at 52.4%), never stopped using WXSIM-Lite since ECMWF Started.
Note : I Run forecasts every 3 hours, which still uses the non updated ECMWF data every other 3rd hour.

https://beaumaris-weather.com/data/wxsimerrimages/

image
image757×503 3.68 KB

P.S. I will note that as per your own words Tommy that Quote: “Precipitation is always a Poo Shoot” and still is even with ECMWF. However Temp and Wind etc. are Bang On.

One last good point is that with ref to "WXSIM - LEARNING ROUTINE RUNS FROM WRET "

" MAE has been equal to pred ( 59 ) Times." :slight_smile:

https://beaumaris-weather.com/historicWxSimMAEDeltaDaily_SA_inc.php

image
image1911×808 116 KB

I also like this Actual Near Realtime vs Forecast page I created. ( Make sure you move your mouse around then center it).
https://beaumaris-weather.com/charts_Combined_Forecast_Actual.php

Kindest Regards,
Tony

4

Hi Steve,

I remember this request, and it’s a good one, but quite complicated, and I haven’t managed to implement it. HOWEVER, I have been researching these things in my own forecasts, and have some results. First, ECMWF does appear to be significantly more accurate than GFS, but not so much so as to make it exclusive. It’s hard to prove, and I really will need to collect more data over time, but my current thinking is that a 70:30 mix of ECMWF:GFS may be close to the “sweet spot”.

As you might suspect, this is likely to have ramifications regarding what weighting of WXSIM-Lite to use. Long studies I’d done in the past led to my advice of about 60-65% WXSIM-Lite weighting. However, that product is derived from GFS, which is, as I said above, a bit less accurate than ECMWF. That would tend to argue toward lowering that 60-65% figure. I just completed a study of 305 forecasts, spanning April 22 through July 11, for each of three scenarios, all will ECMWF:GFS set at 70:30. I tried WXSIM-Lite at 0, 35, and 70% usage. The best fit came at 40% WXSIM-Lite usage, and the differences were fairly small. That makes sense, because the use of ECMWF was already improving the forecasts anyway, and a GFS-derived addition was not going to help as much as it used to (because the forecasts were already better!). However, it did help some (by about 4%), with the 35% choice having a MAE or 2.43 F, and 0% having 2.52, and 70% 2.48.

There’s a subtlety, though. Those numbers are just for max and min temperature forecasts. I also looked at average afternoon and average late night/early morning errors, and in that case, the more WXSIM-Lite, the better (up to 70% anyway). I think I understand this; WXSIM-Lite applies its influence hour by hour, and helps replicate the shape of the diurnal curve. If you’re looking ONLY at max and min temps, the 40% is best. If you also care about getting hourly values right, a higher number would be good. Personally, I think I’m going to go for about 50% WXSIM-Lite.

It’s really good news that this is getting so “nit-picky” as that means the forecasts are already really good with this combo!

I hope that helps! I can definitely share more about how I set up these comparisons if you like.

Tom

5

Hey Tom,

Thanks very much for the thorough explanation. I may ask for more about how you set up the comparisons at some point, but I greatly appreciate this explanation for now. I already have my ECMWF:GFS ratio set to 70:30, but I think I’ll drop my WxSim-Lite Mix from 60 to somewhere between 40 and 50 and see how that goes for a while. My MAE had dropped to 1.91 on June 13, but then rocketed to 3.17 by July 7. It’s currently settled between 2.80 and 2.85 for now. Nothing changed on my system, but the normal Texas heat kicked in. Hopefully, changing my mix a little might drop it back closer to it’s normal summer range between 2.00 and 2.25.

As always, I appreciate the info.

Steve