WU data arrays Update Script ** Closed **

[pwsdashboard]

The name of the topic tells it all.

[ol]
[li]Make a backup of the ./wudata/ folder[/li]
[li]Unzip attached folder[/li]
[li]Place the _wu_upd.php script in your pwsWD??/ folder[/li]
[li]Move the sorttable.js script to your pwsWD??/js/ folder[/li]
[li]Make a copy of one of your pwsWD??/wudata/xxxxx.arr files to test with[/li]
[li]Run the script pwsWD??/_wu_upd.php in your browser[/li]
[li]Check the pdf before posting your questions.[/li][/ol]

Have fun,
Wim

EDIT: 2020-10-29 Updated typo’s - new data-lines can be entered now also

---

wu_arrays_update 3.zip (11.6 KB)

[bitsostring]

@wvdkuil post:1

The name of the topic tells it all.

=D> works in 1907 version, too, once I changed a couple of PWS_ filenames to w34_ =D>

Next question: how do I add missing days?

[pwsdashboard]

@bitsostring post:2

=D> works in 1907 version, too, once I changed a couple of PWS_ filenames to w34_ =D>

Next question: how do I add missing days?

If the data is at WU you could delete the .arr and let the other wu scripts recreate the .arr

No problem to add an “insert 1 day” field, but my question then: If the data is not at WU how do you know what values to add for the missing data?

Wim

[bitsostring]

@wvdkuil post:3

If the data is at WU you could delete the .arr and let the other wu scripts recreate the .arr

No problem to add an “add 1 day” field, but then question: If the data is not at WU how do you know what values to add for the missing data?

No, data is not at WU. I “recreated” 18 days’ relevant data for the wugraphs using manual temp and rain measurements, plus WU readings from a near neighbour’s VP, like this:

2020-02-02,8.1,,2.8,,,,,,,1000.0,998.0,16,,34,0.35

I should have been able to get it from my data logger, but WD doesn’t decode this any more.

No hurry for this extra modification, I’m so glad to get the chance to change existing readings

[pwsdashboard]

@bitsostring post:4

No, data is not at WU. I “recreated” 18 days’ relevant data for the wugraphs using manual temp and rain measurements, plus WU readings from a near neighbour’s VP, like this:

2020-02-02,8.1,,2.8,,,,,,,1000.0,998.0,16,,34,0.35

I should have been able to get it from my data logger, but WD doesn’t decode this any more.

No hurry for this extra modification, I’m so glad to get the chance to change existing readings

Could you test this version.

1. No need to change PWS_ to w34_
2. All missing days should have an empty line with the link to click on.

Wim

---

_wu_upd.php.zip (4.49 KB)

[bitsostring]

@wvdkuil post:5

Could you test this version.

1. No need to change PWS_ to w34_
2. All missing days should have an empty line with the link to click on.

Looking good so far

Question: what (if anything) do I enter for a null/missing reading, so I just get a comma in the .csv file?

[pwsdashboard]

@bitsostring post:6

Looking good so far

Question: what (if anything) do I enter for a null/missing reading, so I just get a comma in the .csv file?

You should mot enter anything for those fields you have no data for.
Those field remains empty as you can see on the two lines here

Old: 
New: 2020-07-03,12,,,,,,,,,,,,,,

Wim

[bitsostring]

@wvdkuil post:5

Could you test this version.

=D> Fantastic :D/

A couple of things:

1. Original version referenced (include_once) PWS/w34_settings.php and PWS/w34_shared.php, but this version doesn’t seem to? Should line 36 read:

$scrpt2 = 'w34_shared.php'

2. Two typos in line 35 (dashboard, script)

3. “Colomn/s” in line 169 (twice) should be “column/s”.

[pwsdashboard]

@bitsostring post:8

. . .

1. Original version referenced (include_once) PWS/w34_settings.php and PWS/w34_shared.php, but this version doesn’t seem to? Should line 36 read:

PWS_ or w34_shared.php was needed for 1 lang function.
I do not like to add all those translations for a script which is not used more then once or twice.
Most non-English speakers will use Google translate as standard.

2. Two typos in line 35 (dashboard, script) 3. "Colomn/s" in line 169 (twice) should be "column/s".

Done, I will replace current version with an updated version in the first post tomorrow.

Wim

[bitsostring]

@wvdkuil post:7

You should not enter anything for those fields you have no data for

One very small thing I noticed: I can add data to an empty field; or change the existing data in a field; but I can not delete existing bad data from a field and leave it blank. I keep getting the message “Data not updated, please try again”.

Luckily the one time I wanted to do that it was an average temp so easy enough to calculate. And it will probably never happen again

[pwsdashboard]

@bitsostring post:10

One very small thing I noticed: I can add data to an empty field; or change the existing data in a field; but I can not delete existing bad data from a field and leave it blank. I keep getting the message “Data not updated, please try again”.

Luckily the one time I wanted to do that it was an average temp so easy enough to calculate. And it will probably never happen again

Yes, I am a programmer by birth. So when a user updates a value, he/she must input a valid value.
Numeric as all values should be. Never even thought about updating a value to “”

Wim

[bitsostring]

@wvdkuil post:11

Never even thought about updating a value to “”

Not a scientist, then? :lol:

It was a day when I only uploaded 2 hours’ data to WU, so I had rubbish max/avg/min values. Luckily I have manual temp/rain records, but I can not correct humidity/dew point. . . they are still wrong. But I only use this WU data for the template graphs, and you have saved me so much time: I have been adding a new line to a corrected master IEDINBUR98Y.txt and uploading it every day. 8O

[Mark]

Thank you for updating the script. I have been waiting for that for a couple of months

I successfully added a missing data for 5th March (whole that day was empty). However, I tried to modify also extreme temperatures and daily rain for 6th March. And I see that .arr file was updated on the server as well as I see new values in the table displayed here: http://pogodaslesin.beep.pl/pogoda/_wu_upd.php
whereas there is no change in the target view: http://pogodaslesin.beep.pl/pogoda/index.php?frame=Reports_at_WU

What could be a reason? Or maybe I will see modified data after the midnight…?

[pwsdashboard]

@Mark post:13

Thank you for updating the script. I have been waiting for that for a couple of months

I successfully added a missing data for 5th March (whole that day was empty). However, I tried to modify also extreme temperatures and daily rain for 6th March. And I see that .arr file was updated on the server as well as I see new values in the table displayed here: http://pogodaslesin.beep.pl/pogoda/_wu_upd.php
whereas there is no change in the target view: http://pogodaslesin.beep.pl/pogoda/index.php?frame=Reports_at_WU

What could be a reason? Or maybe I will see modified data after the midnight…?

BACKGROUND
Almost all script writers cache the data loaded from external servers, also in my scripts, especially for WU data:

1. if it is a previous year the data is not reloaded by the script as one can assume WU is “unable” , or not interested, to update previous years data
2. if it is this years data the data is reloaded once or twice each day and locally cached by the script

This has worked for years until WU decided to stop supplying .CSV files, and stop supplying history data for longer periods then 7 days.

There are two “Man-In-The-Middle” scripts written, 1 by Ken True and 1 by me, to solve this problem.
The MITM script loads data using the new API fromWU in small pieces, assembles those pieces to .CSV data and stores the data in cache-files.
In my MITM script in the .arr files.

“Your problem/ question”
The reports script will re-load this years data tomorrow first time it is run.
Not asking the data from WU but by asking the MITM script for the data in the old format.
You will then see your updated data as the MITM script uses the .arr files to deliver the .CSV the report scripts understands.

When you updated the .arr file for a previous year you have to delete the .CSV file as the reports do not load old years.

Simply delete the cached yearly file in the cache /wureports/wuData/IWIELKOP26-year-2020.txt forcing the reports-script to request the data from the MITM script.

WARNING:
The .arr files are very costly in terms of WU-accesses. Better not delete an .arr file
Also copy the folder when installing a new dashboard version.

Wim

[Mehdi]

@wvdkuil post:1

The name of the topic tells it all.

I’ve got to ask before I get too excited and mess up the data I’ve got from WU for the past 4 years… Do I understand correctly that this would let me edit the WU data and/or enter missing info/days for the data extracted from WU? This would be great, by the way!

Thanks!

[bitsostring]

@Mehdi post:15

Do I understand correctly that this would let me edit the WU data

Not the data held at WU.

and/or enter missing info/days for the data extracted from WU?

Yes. This will let you edit the MITM .arr files stored in /wudata/

But, as Wim says, make a copy first

[Mark]

I modified some data from current year and it works but for 2019 there is still no change.

Simply delete the cached yearly file in the cache /wureports/wuData/IWIELKOP26-year-2020.txt forcing the reports-script to request the data from the MITM script.

I deleted this file as well as …2019.txt and both of them were recreated again but on report site I see old version from WU base. Although my corrections in IWIELKOP26-metric-2019.arr were saved (eg. 5/1/2019 where I changed faulty 25.3*C temperature). Have I missed something?

[pwsdashboard]

@Mark post:17

I modified some data from current year and it works but for 2019 there is still no change.

I deleted this file as well as …2019.txt and both of them were recreated again but on report site I see old version from WU base. Although my corrections in IWIELKOP26-metric-2019.arr were saved (eg. 5/1/2019 where I changed faulty 25.3*C temperature). Have I missed something?

Please clean wudata/ folder.
Do not touch all xxxxxxx.arr files
Rename or delete the current 2019/ folder.

That will solve the mistory.

Problem is that old years data from the caches is not rebuild by the scripts as there was no way to update the WU files at WU.

Now we have the history in the .arr files, we can update the .arr data.
And we then have to clean all 2019 cached .csv files.

Wim

[vilmeteo]

Good morning,

this is my first message, i’m David, from Vila-real in Spain. First of all, congrats for your PWS Dashboard

Yesterday i updated some wrong data form WU using this script, and later, deleted all files in /wudata folder except .arr file.

Today, I see that graphs ara showing the corrected data, but the history is still showing wrong data (month and year rainfall).

You can see it there https://vilmeteo.es/pwsWD/, where rainfall graphs and history data is different.

Thank you!!

[pwsdashboard]

@vilmeteo post:19

Good morning,
this is my first message, i’m David, from Vila-real in Spain. First of all, congrats for your PWS Dashboard
Yesterday i updated some wrong data form WU using this script, and later, deleted all files in /wudata folder except .arr file.
Today, I see that graphs ara showing the corrected data, but the history is still showing wrong data (month and year rainfall).
You can see it there https://vilmeteo.es/pwsWD/, where rainfall graphs and history data is different.
Thank you!!

There are two different sets of “data from the past”
Explained in “my-english” => No High/Lows, where is history and graphs data used for, and ? cron ?

1. The data for the graphs, in your case that comes from WU
   [li]The high-lows for today, yesterday month , year. That is from a separate file.
   You can change the values with . . / pwsWDxx/PWS_hist_update.php?pw=12345
   Replace 12345 with your easyweather password[/li]

Succes,
Wim