ECMWF data changes - Feb 2024

The ECMWF open data source is changing. Yesterday, the directory structure was changed to allow IFS (current) and AIFS (experimental) model data to to included. This broke my download for a short while but that’s now been fixed.

ECMWF have released an updated set of scripts today to handle the IFS/AIFS data, but whilst looking at the new scripts I see they’ve added a ‘resolution’ parameter. Previously the data I processed was at 0.4 degree intervals of latitude and longitude. It appears that this is changing to use 0.25 degree intervals. This would need some code changes in my scripts.

I’m concerned that the new scripts provided by ECMWF only seem to allow a resolution parameter of 0.25 degrees stating that 0.25 is the only resolution available. I’m not sure if that means it’s the only resolution available for the new scripts, or if they’ve already stopped producing 0.4 degree data. If the latter then my scripts will be generating garbage data! I’ve not heard of anyone suggesting that forecasts have gone badly wrong in the last 24 hours so I’m hoping that it’s just not clear wording in the script documentation.

As soon as I can I’ll get a copy of the latest scripts and modify my own scripts to use 0.25 degree data, but until then I’ll just leave the warning out there that ECMWF may become a bit weird in the near future!

1 Like

Looking at the script source code suggests that it’s dropped support for 0.4 degree resolution, but I can also still see 0.4 degree data for download, so the current script is still working properly. I don’t know if I can install the new script alongside the current script though. I’ll have to learn more Python first! If I can’t then I’ll probably have to try to update the ECMWF script, modify my own scripts and test before moving them into production…all within 6 hours!

There’s another complication in that the new scripts are enforcing a newer version of Python than I have available, so I’m going to have to install a new version to run alongside the current version (which can’t be changed because it’s used by the OS).

Sounds like I’m due some fun coding time!

1 Like

Luckily the Python version support hadn’t changed. They’d just added a new supported version rather than setting a new minimum version. So that’s one less problem to worry about.

Now that the 06z run has finished I’ve grabbed the latest ECMWF code and modified my own scripts to support 0.25 degree resolution and some new parameters which need to be passed. I’m doing testing and debugging now.

1 Like

I’ve done a test using today’s 06z run with the new scripts in my dev environment and compared the new data with 06z data produced using the old scripts.

The results are comparable - not identical though. That’s to be expected because data set I’m processing is now a higher resolution which means less interpolation to create the data for each location.

There’s a slight downside to the new data - processing will be a bit slower. There’s more data to download and bigger files to process once downloaded. I don’t think it will make a huge difference though. We’ll have to watch the stats to see what effect it has.

The new scripts will be used for the production database (the data you see) from today’s 12z run (29/02/2024).

2 Likes

The 12z run is failing to download the data. I’m investigating.

1 Like

I’ve fixed the problem.

I’ve parameterised most of my scripts so that they get told whether they’re running in development or production at runtime and they then adjust settnigs/directory paths accordingly. I’ve missed doing one script and that’s one that I had to modify today. It was downloading the files to the development file storage which wasn’t very useful! I’ve fixed the problem temporarily but I’ll need to go back to fix it permanently so that I dont have this problem again.

1 Like

Looking at the stats since I made this change suggests that using 0.25 degree data hasn’t affected the data availability times to WxSimate.

There’s more data to download and unpack, but I suspect there’s also an offset in that 0.4 degree data required significantly more interpolation calculations that aren’t needed for 0.25 degree data.

A significant number of forecast points that are calculated for each run are for 1 degree lat/long intersections (required for WxSim advection). With 0.4 degree data 4 in 5 of these points need interpolating (6 calculations per variable and there are about 15 variables), but 0.25 degree data includes the exact values I need for each 1 degree point, which requires no calculations. That’s about 6 million fewer calculations to do per run.