External scripts which use WeatherUnderground .CSV data

pwsdashboard · 19 June 2020 12:47

Copied from another topic http://discourse.weather-watch.com/p/540878

Added missing screenshot to original post.

Those scripts are examples of scripts which use WU data in the old .CSV format.
They are widely used integrated in stand-alone websites and in template websites using f.i. Saratoga- Leuven- or Meteotemplate
Browsing the forums points you to the download location, for my scripts https://support.leuven-template.eu/

Now your remark “but I donÂ´t see yet how get those scripts into or linked to PWS_Dashboard?”

You can add any stand-alone script in only two, albeit sometimes time-consuming, “steps”
First you have to download and install those scripts “stand-alone” and test them until they are fully functional.
Then you can add a menu entry so the page will be displayed in an iFrame from the menu, such as other iFrame pages.

For the WU scripts using .csv files, there is one advantage:
The needed MITM script to generate the old format data is already included in the PWS_Dashboard.
MITM stands for “Man-In_The_Middle”. It translates the old WU requests to the new WU-api requests storing the data in the cache for future use.

PWS_Dashboard has a script “PWS_DailyHistory.php” which replaces the old WU “WXDailyHistory.asp” script
All those “old” scripts contain one or more lines similar to this one

$wuURLs[] = 'https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID='.$SITE['wu. . . . .

and those lines should now point to the new script

$wuURLs[] = 'https://_change_to_your-site_/_pwsFolder_/PWS_DailyHistory.php?ID='.$SITE['wu. . . . .

As an example I attach a zip of my history script.
Put the unzipped folder wuhistory/ somewhere on your website, but not in the pwsWDxx/ folder, keep it separate.
Run the wuhistory/startWUhistory.php script and adept three lines with your WU startdate, WU-ID and the correct link to PWS_DailyHistory.php.

I will post other WU scripts when time permits.

Wim

Edit 2020-10-31 I included the script also in one of the test-sites => http://pwsdashboard.be/pwsWD/index.php?frame=Reports_at_WU
Edit 2020-11-07 New version, removed missing time-zone, error solar.
Edit 2020-11-08 New version, color switching
Edit 2021-12-11 Removed downloads of pre 8.0 version. New updated version available in 2022

lemuba · 19 June 2020 13:53

Many thanks for the instructions Wim! - I will test/play with it, asap.

Br,

Matthias

pwsdashboard · 20 June 2020 09:22

Attached the folder to use the reports script which allows to compare daily / monthly values for your most important weather-items as stored on WU.

Demo at https://pws.sluispark.be/index.php?frame=Reports_at_WU

Install:
Put the unzipped folder wureports/ somewhere on your website, but not in the pwsWDxx/ folder, keep it separate.
Run the wureports/startReports.php script and adept three lines with your WU startdate, WU-ID and the correct link to PWS_DailyHistory.php.

$wustart 	= '31-05-2029';	  // Important the format is:   dd-mm-YYYY  for your first day of uploading data to WU.      !
$wuid		= 'XXXXXXXXXX';	// your WU station name
$WUdatastr      = 'http://_replace_yourwebsite_/_replace folder_/PWS_DailyHistory.php';

Be aware that the start-date is in the DD-MM-YYYY format which is another format as used in setup script.

To add the scripts to your menu add these lines to _my_settings/frames.php :

# this one is optionally  used if you installed wu history and reports
$show   = true;    
#$show   = false; // remove the # on position 1 if you do not want the wu scripts
if ($show == true) {
        $frame                  = 'History_at_WU';
        $frm_ttls[$frame]       = 'History at WU ';  // name in menu
        $frm_src[$frame]        = 'http://_your_website_/wuhistory/startWUhistory.php';
        $frm_hgth[$frame]       =  1000;  
        $frame                  = 'Reports_at_WU';
        $frm_ttls[$frame]       = 'Reports at WU ';  // name in menu
        $frm_src[$frame]        = 'http://_your_website_/wureports/startReports.php';
        $frm_hgth[$frame]       =  1000;  
        }

Wim

Edit 2021-12-11 Removed downloads of pre 8.0 version. New updated version available in 2022

Mark · 20 June 2020 22:23

I tried to use the script on my website and it worked partially. Colourful reports looks fine: http://mark014.beep.pl/template/plugins/pwsWD/index.php?frame=Reports_at_WU. If I understand correctly, is it not possible to display the temperature with decimal places? It’s a shame.

Unfortunately, I did something wrong with history: http://mark014.beep.pl/template/plugins/pwsWD/index.php?frame=History_at_WU and I have now a message:

Scripts failed, check html for problems

pwsdashboard · 21 June 2020 06:58

I do not like the tone in your remark.
You did not ask if it could display decimals, neither did you check the width, check the data, before telling script-authors that their work need those bashing words.

Feel free to use any script you can lay your hands on, but at least be a little “humble” with your with opinions as long as we can not choose and test those fine scripts you should develop yourself.

Unfortunately, I did something wrong with history: [http://mark014.beep.pl/template/plugins/pwsWD/index.php?frame=History_at_WU](http://mark014.beep.pl/template/plugins/pwsWD/index.php?frame=History_at_WU) and I have now a message:

You did not use the download from the first post. The start script from the download is dated [font=courier][b] 4.01 2020-06-19 Demo ersion for PWS_Dashboard users[/b][/font] Your version is an official stand-alone version downloaded some time ago not already adapted for the PWS_Dashboard. It warns with a message for more experienced users "[color=red]Scripts failed, check html for problems[/color]" and when you check the html you see that you forgot to replace the old link to WeatherUnderground with a link to your MITM script.

data will be loaded from https://www.wunderground.com/weatherstation/WXDailyHistory.asp?

Please delete that folder, download from the first post, upload, change the three lines and it will work.
If you forget to make the changes the message reads “You should adapt lines 11, 12 and 13 before this script can run”

Wim

pwsdashboard · 21 June 2020 07:06

This version is the one using WU.
Jim Murry of WildwoodWeather was the first person ever developing a script like this.
It is mostly used for WeatherDisplay websites using the NOAA reports.
It uses the NOAA/climate data and that data is far more extensive so there are more selectors.
https://weather.wildwoodnaturist.com/downloads.php

Wim

Mark · 21 June 2020 09:41

I am sorry that you understood my utterance in this way. I really appreciate what you are doing and the time spent on developing the script.
There was an error in translating, a nuance of meaning that I did not realize. English is not my native language. To make it faster I put the text in the translator and this time I did not notice such a negative meaning. I meant “what a pity.”

I really did not mean to blame you. I just wish Wunderground sent temperature data with decimal places.

Mark · 21 June 2020 11:31

I exchanged both folders with new ones from the thread and ‘History at WU’ is working now. Thanks for help.

So do you suggest that I should have decimals? I don’t have them.

I made a translation for wureports and feel free to use it if you like.

wulanguage-pl.txt (4.44 KB)

pwsdashboard · 21 June 2020 12:39

I did not suggest that. I said that you should always ask before jumping to conclusions. There is a ton of formatting possibilities in a professional script. The default settings are OK for 80% of the users. Default settings for temperature: fields are rounded to integer and printed without decimals.

But the scripts can cope with your wish. no problem at all.

WARNING: We are now leaving standard installations like when using an app

1. The decimals should be included in your data for temperature
I checked http://mark014.beep.pl/template/plugins/wureports//wuData/IWIELKOP26-year-2020.txt => Seems OK.
But pressure has no decimal with metric only 1 with imperial units.

Date,TemperatureHighC,TemperatureAvgC,TemperatureLowC,DewpointHighC,DewpointAvgC,DewpointLowC,HumidityHigh,HumidityAvg,HumidityLow,PressureMaxhPa,PressureMinhPa,WindSpeedMaxKMH,WindSpeedAvgKMH,GustSpeedMaxKMH,PrecipitationSumCM
2020-01-01,3.1,2.1,0.9,1.4,0.1,-0.6,92,87,84,1035.0,1032.0,31,8,42,0.00
2020-01-02,3.5,0.8,-1.4,-0.3,-2.2,-5.0,96,81,64,1033.0,1027.0,20,6,28,0.00
2020-01-03,5.4,2.9,-0.4,4.2,0.4,-4.2,96,85,67,1027.0,1016.0,28,9,37,0.13

Always make a backup/copy before changing a script!
To display decimals you have to make adjustments in the settings of the script wureports/wsReports1part.php
Line 64 holds the default formatting of the values, it reads now:

$numFormat	= '%01.0f';	// most values from WU are whole numbers

change to

$numFormat	= '%01.1f';	// most values from WU are whole numbers

For the page with temperatures/dewpoint we have set rounding 1 decimal therefor change line 260 from

	$round		= 0;

to

	$round		= 1;

Succes, wim

bitsostring · 21 June 2020 13:19

But if you want decimals and you use round(,1) any decimal “.0” is suppressed, so you get e.g. 13.9, 14, 14.1. . .

Better to use number_format(,1) and get 13.9, 14.0, 14.1.

IMHO

pwsdashboard · 21 June 2020 13:57

That is what I tried to explain. And I discovered also that metric pressure does not need a decimal default, always zero.
So as the only problem is with temperature:
Line 64 holds the default formatting of the values, check if it still has the original:

$numFormat = '%01.0f'; // most values from WU are whole numbers

Line260 reads now

	$round		= 0;

must be changed to round at 1 decimal and the formatting string must be reset to 1 decimal also:

	$round		= 1;
	$numFormat	= '%01.1f';

Looks better at https://pws.sluispark.be/index.php?frame=Reports_at_WU

Wim

bitsostring · 21 June 2020 14:08

Sorry

Mark · 21 June 2020 19:10

Everything now is working fine. Big thanks.

I attached a translation for wuhistory.

I have some errant data in my WU database. If I change some records in wudata files will it be permanent? So the databe isn’t later overwritten?

wulanguage-pl.txt (4.5 KB)

pwsdashboard · 22 June 2020 10:18

The file used for 360 days graph is chartswudata/__wuid__Y.txt and that file is recreated every day from the master cache wudata/.

In the next release I will try, if time permits, to add an update program for past data in the master-cache

Wim

bitsostring · 22 June 2020 10:31

I might be able to adapt that for version 1907

pwsdashboard · 22 June 2020 11:27

That can only be done when there is the MITM script as that script stores its data in a PHP .arr file.
The script uses pws??.wudate/ for the storage of 1 .arr for every year.

Wim

bitsostring · 22 June 2020 11:37

That is what I’m using on version 1907. I am having to add the missing data to the IEDINBUR98Y.txt file, but that gets overwritten every day.

pwsdashboard · 22 June 2020 11:56

The IEDINBUR98Y.txt is not saved by the MITM script.
The file is saved by the w34_load_files.php script into the chartswudata/ folder.

The master-cache for this year is maintained by WXDailyHistory.php and is stored in wudata/IEDINBUR98-metric-2020.arr
This is the data in it https://swanston.maui.co.uk/pws/w34_listfile.php?file=./wudata/IEDINBUR98-metric-2020.arr&type=arr
That array also has the missing data, but is is not overwritten.

The future update script allows to select one day and change/add the fields for that day and it will update the array, not the text-files.

Wim

bitsostring · 22 June 2020 12:17

Thanks, Wim.

Silly question: can I just edit the array manually?

pwsdashboard · 22 June 2020 12:23

Answer is no as it is one long string. Same complexity as a JSON file.
https://swanston.maui.co.uk/pws/wudata/IEDINBUR98-metric-2020.arr

Wim