OpenGrads maps using GRIB2 data

Grads using GRIB2 is possible. Here’s my first attempt at producing a map from existing data. This was hand-driven in the Grads interface but it shows I’m on the right path :slight_smile:

I’ve split this away from the original OpenGrads topic because this isn’t something most people will be able to do so I’ll leave the other thread for the (possibly now broken) OpenGrads maps using OpenDAP (DODS) data.

Looks Brilliant Chris, and sounds brilliant also on the possible mix front. :grinning: :clap:

I’m making progress. I have functional (but not pretty!) scripts that can generate PNG images for temperature at 2m from my already downloaded GFS GRIB2 data. Here’s an example of a 3 hourly cycle covering about 9 days. Be warned, it’s a lot of images so probably not best viewed using mobile data…

This URL might not survive too long, so if you try it in a week or so it might not work, or might show different data.

The images for the global map took less than a minute to create. It’s much quicker to do this when you already have the data downloaded.

I had terrible trouble getting the caching sorted out though. Each time I updated the images the old ones would stick in place. Something to watch out for if you ever have this problem…if you’ve got two domains with similar names make sure you’re adding rules to the correct one :exploding_head:

Looks Brilliant Chris.

A couple more charts to look at

10m wind showing flow direction and speed (colour) - I don’t think this is actually useful, although it could be useful as an art piece. In the style of Van Gogh?

10m wind speed (colours) overlaid with sea level pressure (isobars) - I think this one looks good as an animation…it shows the movement of the high and low pressure areas pretty well and where they cause increased wind speeds. Nice as a visual effect, but they cover too large an area to be useful to anyone locally.

Note: I’m not suggesting these are useful. They’re just demos produced during my testing. I’m aiming at producing charts for smaller areas and with fewer animation steps, e.g. North-West Europe for 48 hours. That’s the next step in my coding work, modify the script to allow it to produce multiple charts for the same variable.

1 Like

Coming right up to present in image formats, I thought I’d look at WEBP (aka weppy). The file sizes are smaller, e.g. for the image below the individual images add up to 49MB but the WEBP image is only 31MB. Also WEBP supports animated images and links to WEBP images are handled nicely by Discourse, for example…

You can click on the image for the original image.

If you’d like to find out more you can install WEBP software onto Linux and convert from PNG (and I’m sure other formats too) into WEBP and there’s also a utility to animate a series of images. There’s probably equivalent software for Windows. I went from reading about how to install WEBP to installing the code, to cnverting some PNGs into WEBP files and and finally animating 70 images to the image you see above in about 30 minutes. I’m happy to share more about how to do this if anyone is interested.

If you’re running an older browser, including Internet Explorer (now out of support/withdrawn?) then the image won’t display for you. About 97% of browsers should display it correctly.

Edit: I created a cut-down animation. Discourse doesn’t display images inline if they’re more than 10MB so to give the proper effect I created an animation with fewer frames.

1 Like

Do you ever have one of those moments where you think “I wonder if that’s possible” and then you try it and wished you’d thought of it sooner?

After a few days of re-scripting to support GRIB2 data files in OpenGrads I’ve basically got things working, as you can see above. I’d been testing with temperature, pressure and wind which had gone well. When I started converting the other parts of the script I discovered that there was a problem with rain data which I couldn’t solve in the way the scripting now worked. That was annoying, but that’s what happens some times.

This morning whilst trying to think how to fix the problem I had one of those “I wonder if that’s possible” moments about the format of the data. I did a quick experiment and it worked. This meant that all the work in the last 4 days was unnecessary. I could simply change one line in the original script and it would work. I’ve made that change now and it does indeed work.

Season 3 Wall GIF by The Simpsons

1 Like

I’m very interested in this. Our meteorological agency provides the forecast in GRIB2 format. I wonder if your scripts are compatible.

GRIB2 is a standard format and OpenGrads will read them so I don’t see why the files couldn’t be used. Having said that the variable names used in GRIB2 files aren’t standardised so you might need to do some translation. GRIB2 files also store multi-dimensional data so the forecast structure might be different.

If the forecast data is accessible online and you can point me at it I can take a quick look to see how similar it is to give you an idea of how much you might need to modify the script.

This is the page that contains the raw data:

Either the first or the second dataset.

I’ve only done a quick check but this is what I’ve found.

All the files are in GRIB1 format rather than GRIB2. That shouldn’t be a problem because OpenGrads can handle both GRIB1 and GRIB2 files. The first link files are very cut down, just temperature @ 2m and wind vectors for each hour for a 6 hourly period. If you only want temp/wind then that’s OK. The second link files are more complete, including 28 variables for 72 hours, i.e.

1:0:d=23101506:PRES:kpds5=1:kpds6=1:kpds7=0:TR=0:P1=67:P2=0:TimeU=1:sfc:67hr fcst:NAve=0
2:16550:d=23101506:TCDC:kpds5=71:kpds6=1:kpds7=0:TR=0:P1=67:P2=0:TimeU=1:sfc:67hr fcst:NAve=0
3:33100:d=23101506:TMP:kpds5=11:kpds6=105:kpds7=2:TR=0:P1=67:P2=0:TimeU=1:2 m above gnd:67hr fcst:NAve=0
4:49650:d=23101506:VGRD:kpds5=34:kpds6=105:kpds7=10:TR=0:P1=67:P2=0:TimeU=1:10 m above gnd:67hr fcst:NAve=0
5:66200:d=23101506:UGRD:kpds5=33:kpds6=105:kpds7=10:TR=0:P1=67:P2=0:TimeU=1:10 m above gnd:67hr fcst:NAve=0
6:82750:d=23101506:PRMSL:kpds5=2:kpds6=102:kpds7=0:TR=0:P1=67:P2=0:TimeU=1:MSL:67hr fcst:NAve=0
7:99300:d=23101506:RH:kpds5=52:kpds6=105:kpds7=2:TR=0:P1=67:P2=0:TimeU=1:2 m above gnd:67hr fcst:NAve=0
8:115850:d=23101506:GP:kpds5=6:kpds6=100:kpds7=925:TR=0:P1=67:P2=0:TimeU=1:925 mb:67hr fcst:NAve=0
9:132400:d=23101506:UGRD:kpds5=33:kpds6=100:kpds7=925:TR=0:P1=67:P2=0:TimeU=1:925 mb:67hr fcst:NAve=0
10:148950:d=23101506:VGRD:kpds5=34:kpds6=100:kpds7=925:TR=0:P1=67:P2=0:TimeU=1:925 mb:67hr fcst:NAve=0
11:165500:d=23101506:TMP:kpds5=11:kpds6=100:kpds7=925:TR=0:P1=67:P2=0:TimeU=1:925 mb:67hr fcst:NAve=0
12:182050:d=23101506:RH:kpds5=52:kpds6=100:kpds7=925:TR=0:P1=67:P2=0:TimeU=1:925 mb:67hr fcst:NAve=0
13:198600:d=23101506:GP:kpds5=6:kpds6=100:kpds7=850:TR=0:P1=67:P2=0:TimeU=1:850 mb:67hr fcst:NAve=0
14:215150:d=23101506:UGRD:kpds5=33:kpds6=100:kpds7=850:TR=0:P1=67:P2=0:TimeU=1:850 mb:67hr fcst:NAve=0
15:231700:d=23101506:VGRD:kpds5=34:kpds6=100:kpds7=850:TR=0:P1=67:P2=0:TimeU=1:850 mb:67hr fcst:NAve=0
16:248250:d=23101506:TMP:kpds5=11:kpds6=100:kpds7=850:TR=0:P1=67:P2=0:TimeU=1:850 mb:67hr fcst:NAve=0
17:264800:d=23101506:RH:kpds5=52:kpds6=100:kpds7=850:TR=0:P1=67:P2=0:TimeU=1:850 mb:67hr fcst:NAve=0
18:281350:d=23101506:GP:kpds5=6:kpds6=100:kpds7=700:TR=0:P1=67:P2=0:TimeU=1:700 mb:67hr fcst:NAve=0
19:297900:d=23101506:UGRD:kpds5=33:kpds6=100:kpds7=700:TR=0:P1=67:P2=0:TimeU=1:700 mb:67hr fcst:NAve=0
20:314450:d=23101506:VGRD:kpds5=34:kpds6=100:kpds7=700:TR=0:P1=67:P2=0:TimeU=1:700 mb:67hr fcst:NAve=0
21:331000:d=23101506:TMP:kpds5=11:kpds6=100:kpds7=700:TR=0:P1=67:P2=0:TimeU=1:700 mb:67hr fcst:NAve=0
22:347550:d=23101506:RH:kpds5=52:kpds6=100:kpds7=700:TR=0:P1=67:P2=0:TimeU=1:700 mb:67hr fcst:NAve=0
23:364100:d=23101506:GP:kpds5=6:kpds6=100:kpds7=500:TR=0:P1=67:P2=0:TimeU=1:500 mb:67hr fcst:NAve=0
24:380650:d=23101506:UGRD:kpds5=33:kpds6=100:kpds7=500:TR=0:P1=67:P2=0:TimeU=1:500 mb:67hr fcst:NAve=0
25:397200:d=23101506:VGRD:kpds5=34:kpds6=100:kpds7=500:TR=0:P1=67:P2=0:TimeU=1:500 mb:67hr fcst:NAve=0
26:413750:d=23101506:TMP:kpds5=11:kpds6=100:kpds7=500:TR=0:P1=67:P2=0:TimeU=1:500 mb:67hr fcst:NAve=0
27:430300:d=23101506:RH:kpds5=52:kpds6=100:kpds7=500:TR=0:P1=67:P2=0:TimeU=1:500 mb:67hr fcst:NAve=0
28:446850:d=23101506:APCP:kpds5=61:kpds6=1:kpds7=0:TR=4:P1=0:P2=67:TimeU=1:sfc:0-67hr acc:NAve=0

The existing script can handle more data than is in these files, but there’s enough useful data in these files, e.g. pressure at surface and MSL, temperature at 2m, 925, 850, 700 and 500mb, wind vectors at 10m, 925, 850, 700 and 500mb, relative humidity at 2m, 925, 850, 700 and 500mb, total cloud cover, accumulating rainfall and geopotential at 925, 850, 700 and 500mb. The downside of the second link files is that they’re much bigger to download.

The variable names look good too so there’s a good chance that some charts can be made from the files. I don’t remember if you’re a Linux or Windows user, but if Linux then as a starting point I’d suggest:

  1. Download the wgrib source code (from Wesley’s page on the NWS web site) and compile it. This will likely need some additional compiler tools and libraries installing so it’s probably not going to be a quick process.

  2. Install the latest version of OpenGrads (if I remember correctly it’s on GitHub).

  3. Download/compile grib2ctl from the NWS web site. You’ll also need gribmap which I think is included in OpenGrads, but if not, source it and compile/install it.

  4. When you get this far you’ll need to download one of the forecast ZIP files and extract it.

  5. First step is to check that wgrib works OK, so use “wgrib name_of_one_of_the_files.grb” and see if you get some output like above.

  6. If you do, then you need to create the control and index files for the GRIB1 file. Use grib2ctl and gribmap to do this. I use alternative tools for GRIB2 working so I’m not sure how the GRIB1 tools work. Possibly something like “grib2ctl your_grib_file.grib > your_grib_file.ctl” and “gribmap -i your_grib_file.ctl”.

  7. Now see if OpenGrads can open the file. Open grads in batch command line more using “grads -lcb”. You should get a GA-> prompt. If so, try entering “'open your_grib_file.ctl”. Note that there’s a single quote ’ before the open. If that works type ‘q ctlinfo’ to check that OpenGrads has been able to make sense of the file contents.

  8. If you get to this point, post the output from “q ctlinfo” here and I’ll take a look and suggest how to proceed with the scrpit to generate some chart images.

Thanks for all the details, Chris. I’ll start at step 1 and see how it goes. I think I tried to do this a while ago but got stuck at some point.

I made it all the way through. After step 5 I got the same output as you did.
I created the test.ctl file (filesize 624 B).
The issue I have is I don’t get any output from the last command:

[19] ga-> "'open test.ctl"
Out[19]: "'open test.ctl"

[20] ga-> "q ctlinfo"
Out[20]: 'q ctlinfo'

Any hint perhaps?

The double quotes are probably causing the problem. Use:

'open test.ctl'
'q ctlinfo'

Sorry for causing confusion.

I think it must be something else. I tried using single quotes but no success. Do you think it is related to file-path?
If the .ctl file was corrupt or incompatible, I should probably get an error message, right?

The .ctl file should be text so you can read it and put a copy here if you like. For now, don’t worry about the path…put the .grb in a directory. Then, making sure you’re in that directory, create the .ctl and .idx files using grib2ctl and gribmap. Then run grads from the same directory.

The directory path can get encoded into the .ctl file so that might be what’s causing the problem.

This is the content of my .ctl file:

dset ^public_inca_20231015-1800.grb
index ^public_inca_20231015-1800.grb.idx
undef 9.999E+20
title public_inca_20231015-1800.grb
*  produced by grib2ctl v0.9.15
* command line options: -verf public_inca_20231015-1800.grb
dtype grib 255
pdef 401 301 lcc 44.688000 12.235000 1 1 46.120000 46.120000 14.815000 1000 1000
xdef 401 linear 12.235000 0.0127870459321199
ydef 301 linear 44.688000 0.00909090909090909
tdef 7 linear 18Z15oct2023 1hr
zdef 1 linear 1 1
vars 3
TMP2m  0 11,105,2 ** 2 m above ground Temp. [K]
UGRD10m  0 33,105,10 ** 10 m above ground u wind [m/s]
VGRD10m  0 34,105,10 ** 10 m above ground v wind [m/s]

Directory content:


Visually that looks OK but I’m not at my laptop so I can’t compare with my own examples. I have a few chores to do this morning but I’ll take a closer look later on.

Here’s what I just tried.

I have the .grb file and in the same directory. I did…

./ nwp_2023101506_0003.grb >nwp3.ctl

That creates the nwp3.ctl control file. Then I did:

~/grads/xx # gribmap -i nwp3.ctl
gribmap: opening GRIB file nwp_2023101506_0003.grb
gribmap: reached end of files
gribmap: writing the GRIB1 index file (version 5)

This creates the nwp_2023101506_0003.grb.idx file. Next run ‘grads -lcb’ to get the ga-> prompt. This is where I think the problem is…in scripts you have to surround some stuff with single quotes. You don’t need this at the interactive prompt, so…

~/grads/xx # grads -lcb

              Welcome to the OpenGrADS Bundle Distribution

For additional information enter "grads --manual".

Starting "/opt/opengrads/Linux/Versions/2.2.1.oga.1/x86_64/grads   -lcb " ...

Note: -c option was specified, but no command was provided

Grid Analysis and Display System (GrADS) Version 2.2.1.oga.1
Copyright (C) 1988-2018 by George Mason University
See file COPYRIGHT for more information

Config: v2.2.1.oga.1 little-endian readline grib2 netcdf hdf4-sds hdf5 opendap-grids,stn athena geotiff shapefile
Issue 'q config' and 'q gxconfig' commands for more detailed configuration information
Loading User Defined Extensions table </opt/opengrads/Linux/Versions/2.2.1.oga.1/x86_64/gex/udxt> ... ok.
GX Package Initialization: Size = 11 8.5
Running in Batch mode
ga-> open nwp3.ctl
Scanning description file:  nwp3.ctl
Data file nwp_2023101506_0003.grb is open as file 1
LON set to 11.625 17.895
LAT set to 44.642 47.442
LEV set to 925 925
Time values set: 2023:10:15:6 2023:10:15:6
E set to 1 1
ga-> q ctlinfo
dset nwp_2023101506_0003.grb
title nwp_2023101506_0003.grb
undef 9.999e+20
xdef 111 linear 11.625 0.057
ydef 71 linear 44.642 0.04
zdef 4 levels 925 850 700 500
tdef 1 linear 06Z15OCT2023 1mo
vars 13
apcpsfc  0  61  ** surface Total precipitation [kg/m^2]
gpprs  4  6  ** (profile) Geopotential [m^2/s^2]
pressfc  0  1  ** surface Pressure [Pa]
prmslmsl  0  2  ** mean-sea level Pressure reduced to MSL [Pa]
rhprs  4  52  ** (profile) Relative humidity [%]
rh2m  0  52  ** 2 m above ground Relative humidity [%]
tcdcsfc  0  71  ** surface Total cloud cover [%]
tmpprs  4  11  ** (profile) Temp. [K]
tmp2m  0  11  ** 2 m above ground Temp. [K]
ugrdprs  4  33  ** (profile) u wind [m/s]
ugrd10m  0  33  ** 10 m above ground u wind [m/s]
vgrdprs  4  34  ** (profile) v wind [m/s]
vgrd10m  0  34  ** 10 m above ground v wind [m/s]

Hopefully you can replicate this.