Getting Data from Another Webpage?

I have a records page I am doing for my site: http://www.dfwweather.org/wxrecords2.php

As an example, I would like to pull the daily low temperature from this site http://www.wunderground.com/history/airport/KDFW/2009/4/1/DailyHistory.html and display it in the appropriate column on my page (link above).

If it is not possible to do, how would one go about getting the daily climate summary from the National Weather Service and parse it, extract the pertinent data needed, and display it using php to essentially get the same result as above? Still very new to php.

Any help would greatly be appreciated.

Perhaps I am making this too hard?

Here is a script that will take your wunderground stats…

http://jcweather.us/scripts.php

will look like this…

http://jcweather.us/history.php

Hope this helps.

Thanks for your reply. I really just need to extract the specific low temperature value and display that value on my page as I want to stick to the table and formatting I’ve already desinged.

Any suggestions on how to do that or will the script you provided do that with some modification?

i dont think your going to be able just to get THAT value using a script. So you dont want all that other data in that script?

No, don’t need all that other data. You can see the table I already designed here at www.dfwweather.org/wxrecords2.php.

The record data already listed in the table works which I display from text files containing that data on my server. Another possible way to get that data is from the daily climate report from the NWS, which is a text file. However, I’m not sure how to get that uploaded to my site via WD. Also, I’ve tried to parse it and extract the pertinent data needed, but my limited PHP skills leave me lost. Weatherunderground somehow gets the data from the NWS daily climate report as notated at the bottom of their page. So, not sure how they are doing it either. I just thought it might be easier to somehow parse the weatherunderground page at the link I gave above and extract the data that was needed. Maybe that is not doable from a webpage?

WD can do a climate data page off of your own data like this…

http://www.snoqualmieweather.com/wxclimate.php

But Im not sure how your going to get just the data you want from another page. Maybe someone here that writes scripts could give you more of an idea how to go about that.

That’s the problem though, I don’t have a weather station, so my site relies on the metar data from DFW airport. I want my record page that I have designed to display the official records for the DFW area which is taken from DFW airport.

Oh…ok… Now i understand what your trying to do. I didnt realize you didnt have a station.

Didn’t Ken already answer this question here?

That looks like exactly what he wants to do. Maybe he forgot he posted about that, and never checked back on it.

Ken answered another question. He just told me how to display the record max and min temp and year that it occurred in that post, which I already have working. I am looking for the actual low temperature for the current day at DFW airport in this thread. Ultimately, I want yesterday’s low and high, the current low and high for the current day, and the same for precipitation to be displayed as officially recorded from DFW airport. Everything else on my page is working. So, my question is what is the best way to get that data? It is found in two places that I know of, the National Weather Service daily climate report, or on this page at weatherunderground: http://www.wunderground.com/history/airport/KDFW/2009/4/1/DailyHistory.html?req_city=NA&req_state=NA&req_statename=NA

What I want is for it to display in my table here: www.dfwweather.org/wxrecords2.php

One of many different ways…

<?php

// What is the URL you are going to get
$URL =  'http://www.wunderground.com/history/airport/KDFW/2009/4/1/DailyHistory.' .
        'html?req_city=NA&req_state=NA&req_statename=NA';
        
// Obtain page into an array
$raw_string = file($URL);


// Search through the array to find the line that has that in it
// There are a number of different ways of doing this including using
// array_search etc...

echo "Max = " . get_value("Max Temperature") . "
";
echo "Min = " . get_value("Min Temperature") . "
";
exit;

function get_value ($lookfor) {
    global $raw_string;

    foreach($raw_string as $key => $value ) {
        
        // Does the string exist in this this element?
        if ( strpos($value,$lookfor) !== FALSE ) {
            
            // Yep, so the value is in two array elements further
            
            // Find where the data is
            
            // Starts
            $ptr1 = strpos($raw_string[$key + 2],'class="b">') + 10;
            
            // Ends
            $ptr2 = strpos($raw_string[$key + 2], '</', $ptr1) - $ptr1 ;
            
            // Output the data we found
            return (substr($raw_string[$key + 2],$ptr1,$ptr2));
        }
    } 
}


You should cache this info since it won’t change often and takes a while to obtain.

Example of same code working: csc0709-01.php

Thanks for your reply, but I’m not following.

First, you say cache the info, I don’t know how to do that. Remember, you are really talking to a beginner in php.

Second, let’s just concentrate on one value for now - say the current day’s low temp. I thinkg I will be able to go from there - let’s hope!

I’ve noticed that the URL for weatherunderground contains the date in the URL, so somehow the code is going to have to be dependent on the current date - right? This is so it will automatically update for each new day.

Now, I tried your code and did get the current value for the low temp for today (4/1/2009), however, the rest of the code for the table truncated after that value. So, the record data never appeared nor did the rest of my table. Not sure what happened there?

First things first…

Second, let's just concentrate on one value for now - say the current day's low temp. I thinkg I will be able to go from there - let's hope!

The script just showed how to get the values and displayed them on the screen.

Other values may not be as easy and may require other coding depending on how it is in the page.

Using the code, you can get the value of either value (Min or Max) with the function:

get_value(“Min Temperature”);

which would return the value of the MIN value.

or

get_value(“Max Temperature”);

which would return the value of the MAX value.

I've noticed that the URL for weatherunderground contains the date in the URL, so somehow the code is going to have to be dependent on the current date - right? This is so it will automatically update for each new day.

This can be fixed by changing the URL code like:

<?php 
// Get current date
$cur_date = date("o/n/j");

// What is the URL you are going to get
$URL =  'http://www.wunderground.com/history/airport/KDFW/' . 
        $cur_date . 
        '/DailyHistory.html?req_city=NA&req_state=NA&req_statename=NA';

This obtains the current date from the server and inserts it into the URL that will be used to get the data.

Now, I tried your code and did get the current value for the low temp for today (4/1/2009), however, the rest of the code for the table truncated after that value. So, the record data never appeared nor did the rest of my table. Not sure what happened there?

Hard to really tell what is going on without seeing how you attempted to implement it. The code provided above just showed how to get the value rather than how to integrated into your existing code which I’ve not seen.

If you post a copy of the script, we can see how you are attempting to use the code.

Here is my code for my page. I am using Dreamweaver CS3 and I know it is a bit rough, but please bear with me I am learning. It seems when I put in your code where I want the value to appear in the table it works, but the rest of my table is truncated and it ignores any other PHP that I have in the table. Hope that makes sense!


<?php
############################################################################
# A Project of TNET Services, Inc. and Saratoga-Weather.org (WD-USA template set)
############################################################################
#
#   Project:    Sample Included Website Design
#   Module:     sample.php
#   Purpose:    Sample Page
#   Authors:    Kevin W. Reed <[email protected]>
#               TNET Services, Inc.
#
# 	Copyright:	(c) 1992-2007 Copyright TNET Services, Inc.
############################################################################
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA
############################################################################
#	This document uses Tab 4 Settings
############################################################################
require_once("Settings.php");
require_once("common.php");
############################################################################
$TITLE= $SITE['organ'] . " - Daily Records";
$showGizmo = true;  // set to false to exclude the gizmo
include("top.php");
############################################################################
?>
</head>
<style type="text/css">
<!--
.style1 {
	color: #FFFFFF;
	font-weight: bold;
	font-size: 14px;
}
.style2 {
	font-size: 9px;
	font-weight: bold;
}
#main-copy table {
	border-top-width: thin;
	border-right-width: thin;
	border-bottom-width: thin;
	border-left-width: thin;
	border-top-style: solid;
	border-right-style: solid;
	border-bottom-style: solid;
	border-left-style: solid;
}
.style7 {
	font-size: 12px;
	font-weight: bold;
}
.style8 {font-size: 12px; }
#main-copy td {
}
-->
</style>
<body>
<?php
############################################################################
include("header.php");
############################################################################
include("menubar.php");
############################################################################
?>
<?php
	// select this months numeric value
	$thismonth = date("n");
	//select file to open
	$myFile = "./Records/".$thismonth.".txt";
	// open file
	$fh = fopen($myFile, 'r');
	// load file data into a variable for handling later
	$theData = fread($fh, filesize($myFile));
	// close file
	fclose($fh);

	// select day for row search
	$thisday = date("j");
	//split data file into an array so we can walk around in it \r\n means split contents on new lines we do this so we can loop through each row of the array performing actions as required.
	$lookuparray = explode("\n",$theData);
	// get a count of array length
	$arraycount = count($lookuparray);
?>

<div id="main-copy">  
  <p>
  <h1 align="left">DFW Weather Records/Daily Summary</h1>
  <p align="left">(As officially recorded from DFW Airport.  Period of record began in September 1898.)</p>
  <table width="661" border="1" cellpadding="2" cellspacing="0" bordercolor="#666666">
    
    <tr bgcolor="#000099">
      <td colspan="9" bgcolor="#3173b1"><div align="left"><span class="style1">Daily Summary</span></div></td>
    </tr>
    <tr>
      <td colspan="9"><strong><? print(Date("F j, Y")); ?></strong></td>
    </tr>
    <tr>
      <td width="93" rowspan="2" bgcolor="#666666">&nbsp;</td>
      <td colspan="2"><div align="center" class="style8"><strong>Yesterday</strong></div>        <div align="center" class="style8"></div></td>
      <td colspan="2"><div align="center" class="style8"><strong>Today</strong></div>        <div align="center" class="style8"></div></td>
      <td colspan="2"><div align="center" class="style8"><strong>Normal</strong></div>        <div align="center" class="style8"></div></td>
      <td colspan="2"><div align="center" class="style8"><strong>Record</strong></div>        <div align="center" class="style8"></div></td>
    </tr>
    <tr>
      <td width="55"><div align="center" class="style2">Low</div></td>
      <td width="55"><div align="center" class="style2">High</div></td>
      <td width="55"><div align="center" class="style2">Low</div></td>
      <td width="55"><div align="center" class="style2">High</div></td>
      <td width="55"><div align="center" class="style2">Low</div></td>
      <td width="55"><div align="center" class="style2">High</div></td>
      <td width="84"><div align="center" class="style2">Low</div></td>
      <td width="84"><div align="center" class="style2">High</div></td>
    </tr>
    <tr>
      <td><div align="right"><strong>Temperature:</strong></div></td>
      <td><div align="center"><span style="text-align: center;">
        <?php 
			               echo strip_units($mintempyest) . $uomTemp; ?>
      </span></div></td>
      <td><div align="center"><span style="text-align: center;">
        <?php 
			            echo strip_units($maxtempyest) . $uomTemp; ?>
      </span></div></td>
      <td><div align="center"><span class="ajax">
		<?php

// What is the URL you are going to get
$URL =  'http://www.wunderground.com/history/airport/KDFW/2009/4/1/DailyHistory.' .
        'html?req_city=NA&req_state=NA&req_statename=NA';
        
// Obtain page into an array
$raw_string = file($URL);


// Search through the array to find the line that has that in it
// There are a number of different ways of doing this including using
// array_search etc...

echo get_value("Min Temperature") . "&deg;F";
exit;

function get_value ($lookfor) {
    global $raw_string;

    foreach($raw_string as $key => $value ) {
        
        // Does the string exist in this this element?
        if ( strpos($value,$lookfor) !== FALSE ) {
            
            // Yep, so the value is in two array elements further
            
            // Find where the data is
            
            // Starts
            $ptr1 = strpos($raw_string[$key + 2],'class="b">') + 10;
            
            // Ends
            $ptr2 = strpos($raw_string[$key + 2], '</', $ptr1) - $ptr1 ;
            
            // Output the data we found
            return (substr($raw_string[$key + 2],$ptr1,$ptr2));
        }
    } 
} ?>
      </span></div></td>
      <td width="55"><div align="center"><span class="ajax">
	  
		<?php

// What is the URL you are going to get
$URL =  'http://www.wunderground.com/history/airport/KDFW/2009/4/1/DailyHistory.' .
        'html?req_city=NA&req_state=NA&req_statename=NA';
        
// Obtain page into an array
$raw_string = file($URL);


// Search through the array to find the line that has that in it
// There are a number of different ways of doing this including using
// array_search etc...

echo get_value2("Max Temperature") . "&deg;F";
exit;

function get_value2 ($lookfor) {
    global $raw_string;

    foreach($raw_string as $key => $value ) {
        
        // Does the string exist in this this element?
        if ( strpos($value,$lookfor) !== FALSE ) {
            
            // Yep, so the value is in two array elements further
            
            // Find where the data is
            
            // Starts
            $ptr1 = strpos($raw_string[$key + 2],'class="b">') + 10;
            
            // Ends
            $ptr2 = strpos($raw_string[$key + 2], '</', $ptr1) - $ptr1 ;
            
            // Output the data we found
            return (substr($raw_string[$key + 2],$ptr1,$ptr2));
        }
    } 
} ?> ?></span></div></td>
      <td><div align="center">
      <?php
	// loop through array starting at the start in arrays thats element zero
	for($i=0;$i<$arraycount;$i++){
	// if we hit the day we want do something
	if ($i=$thisday) {
	// split the retrieved line matching the current day for processing.
	$perlinearray = explode("\t",$lookuparray[$i]);
	// echo out all 11 columns you can work out what to do with it later or if you need to select only a specific column you can use the principals above to work it out :)
	echo $perlinearray[9];
	// end processing after displaying the line we want
	break;
	} else { 
	echo " ";
	}
	}
	?>&deg;F</div></td>
      <td><div align="center">
      <?php
	// loop through array starting at the start in arrays thats element zero
	for($i=0;$i<$arraycount;$i++){
	// if we hit the day we want do something
	if ($i=$thisday) {
	// split the retrieved line matching the current day for processing.
	$perlinearray = explode("\t",$lookuparray[$i]);
	// echo out all 11 columns you can work out what to do with it later or if you need to select only a specific column you can use the principals above to work it out :)
	echo $perlinearray[10];
	// end processing after displaying the line we want
	break;
	} else { 
	echo " ";
	}
	}
	?>&deg;F</div></td>
      <td><div align="center">
      <?php
	// loop through array starting at the start in arrays thats element zero
	for($i=0;$i<$arraycount;$i++){
	// if we hit the day we want do something
	if ($i=$thisday) {
	// split the retrieved line matching the current day for processing.
	$perlinearray = explode("\t",$lookuparray[$i]);
	// echo out all 11 columns you can work out what to do with it later or if you need to select only a specific column you can use the principals above to work it out :)
	echo $perlinearray[1];
	// end processing after displaying the line we want
	break;
	} else { 
	echo " ";
	}
	}
	?>&deg;F (<?php
	// loop through array starting at the start in arrays thats element zero
	for($i=0;$i<$arraycount;$i++){
	// if we hit the day we want do something
	if ($i=$thisday) {
	// split the retrieved line matching the current day for processing.
	$perlinearray = explode("\t",$lookuparray[$i]);
	// echo out all 11 columns you can work out what to do with it later or if you need to select only a specific column you can use the principals above to work it out :)
	echo $perlinearray[2];
	// end processing after displaying the line we want
	break;
	} else { 
	echo " ";
	}
	}
	?>)</div></td>
      <td width="84"><div align="center">
      <?php
	// loop through array starting at the start in arrays thats element zero
	for($i=0;$i<$arraycount;$i++){
	// if we hit the day we want do something
	if ($i=$thisday) {
	// split the retrieved line matching the current day for processing.
	$perlinearray = explode("\t",$lookuparray[$i]);
	// echo out all 11 columns you can work out what to do with it later or if you need to select only a specific column you can use the principals above to work it out :)
	echo $perlinearray[3];
	// end processing after displaying the line we want
	break;
	} else { 
	echo " ";
	}
	}
	?>&deg;F (<?php
	// loop through array starting at the start in arrays thats element zero
	for($i=0;$i<$arraycount;$i++){
	// if we hit the day we want do something
	if ($i=$thisday) {
	// split the retrieved line matching the current day for processing.
	$perlinearray = explode("\t",$lookuparray[$i]);
	// echo out all 11 columns you can work out what to do with it later or if you need to select only a specific column you can use the principals above to work it out :)
	echo $perlinearray[4];
	// end processing after displaying the line we want
	break;
	} else { 
	echo " ";
	}
	}
	?>)</div></td>
    </tr>
    <tr>
      <td bgcolor="#666666">&nbsp;</td>
      <td colspan="2"><div align="center" class="style7">Yesterday</div></td>
      <td colspan="2"><div align="center" class="style7">Today</div></td>
      <td colspan="2"><div align="center" class="style7">Record</div></td>
      <td colspan="2" rowspan="3" bgcolor="#666666">&nbsp;</td>
    </tr>
    <tr>
      <td><div align="right"><strong>Rain:</strong></div></td>
      <td colspan="2"><div align="center"><span class="ajax"><?php echo strip_units($yesterdayrain) . $uomRain; ?></span></div></td>
      <td colspan="2"><div align="center"><span class="ajax"><?php echo strip_units($dayrn) . $uomRain; ?></span></div></td>
      <td colspan="2"><div align="center">
      <?php
	// loop through array starting at the start in arrays thats element zero
	for($i=0;$i<$arraycount;$i++){
	// if we hit the day we want do something
	if ($i=$thisday) {
	// split the retrieved line matching the current day for processing.
	$perlinearray = explode("\t",$lookuparray[$i]);
	// echo out all 11 columns you can work out what to do with it later or if you need to select only a specific column you can use the principals above to work it out :)
	echo $perlinearray[5];
	// end processing after displaying the line we want
	break;
	} else { 
	echo " ";
	}
	}
	?>&quot; (<?php
	// loop through array starting at the start in arrays thats element zero
	for($i=0;$i<$arraycount;$i++){
	// if we hit the day we want do something
	if ($i=$thisday) {
	// split the retrieved line matching the current day for processing.
	$perlinearray = explode("\t",$lookuparray[$i]);
	// echo out all 11 columns you can work out what to do with it later or if you need to select only a specific column you can use the principals above to work it out :)
	echo $perlinearray[6];
	// end processing after displaying the line we want
	break;
	} else { 
	echo " ";
	}
	}
	?>)</div></td>
    </tr>
    <tr>
      <td><div align="right"><strong>Snow:</strong></div></td>
      <td colspan="2"><div align="center">0.00&quot;</div></td>
      <td colspan="2"><div align="center">0.00&quot;</div></td>
      <td colspan="2"><div align="center">
          <?php
	// loop through array starting at the start in arrays thats element zero
	for($i=0;$i<$arraycount;$i++){
	// if we hit the day we want do something
	if ($i=$thisday) {
	// split the retrieved line matching the current day for processing.
	$perlinearray = explode("\t",$lookuparray[$i]);
	// echo out all 11 columns you can work out what to do with it later or if you need to select only a specific column you can use the principals above to work it out :)
	echo $perlinearray[7];
	// end processing after displaying the line we want
	break;
	} else { 
	echo " ";
	}
	}
	?> 
          (<?php
	// loop through array starting at the start in arrays thats element zero
	for($i=0;$i<$arraycount;$i++){
	// if we hit the day we want do something
	if ($i=$thisday) {
	// split the retrieved line matching the current day for processing.
	$perlinearray = explode("\t",$lookuparray[$i]);
	// echo out all 11 columns you can work out what to do with it later or if you need to select only a specific column you can use the principals above to work it out :)
	echo $perlinearray[8];
	// end processing after displaying the line we want
	break;
	} else { 
	echo " ";
	}
	}
	?>)</div></td>
    </tr>
  </table>
</div>
<!-- end main-copy -->

<?php
############################################################################
include("footer.php");
############################################################################
# End of Page
############################################################################
?> 

The table is supposed to look like this www.dfwweather.org/wxrecords3.php
The data in the normal and record fields under temperatures are correct, as well as under the rain and snow record field. I just need to basically get the info for the yesterday and today fields. If I can get one, such as the Low under today to work, I think I can figure out the rest.

I won’t be able to really look at this until later today. Busy day…

Some comments though:

o Hard to learn HTML when you use something like Dreamweaver… While Dreamweaver “helps” you see what things look like, it creates messy code, doesn’t force you to learn the basics and you are much better of diving in without it if learning HTML, CSS and PHP is the goal.

Most of the templates are almost plug an play and using just a plain text editor, ftp client and browser to show you what you are doing will help you learn faster.

o You should use CSS instead of embedding codes into your HTML ie:

<td colspan="2"><div [b]align="center"[/b] [b]class="style8"[/b]>[b]<strong>[/b]Record[b]</strong>[/b]</div>[color=red]<div [b]align="center"[/b] [b]class="style8[/b]"></div>[/color]</td>

This might be Dreamweaver doing this but it is really gross, harder to keep track of and may cause browser issues with different browsers as they lose track of what to promote.

Example, the last section in red doesn’t appear to actually do anything as there is no displayable data between the

's

Yeah, no what you mean about busy.

I actually know HTML and Flash fairly well, but problem is I haven’t really done hard web programming or otherwise since college and lot as changed in the last 10 years. I can basically figure out how do all the modules if I am shown how to do one, like the low (I think). My ultimate goal is not to learn PHP well, but enough to accomplish what I need done, and to get my records page to be fully automated. Currently, I have to manually update yesterday’s high, low, and precipitation. I admit I use Dreamweaver cause I like seeing it, and I know it kicks out rudimentary code, same for FrontPage back in the day.

Another question for you is for the data I want to get displayed on my site, is it best pulling it directly from that URL like were doing? Although, I don’t know where else to pull it from other than the NWS Daily Climate Report, but how I get that text file to my server is beyond me too? I would also like to understand a little more about the caching thing you suggested earlier.

Again, many thanks for your help.

The problem is that you took the script, and then inserted it into your code multiple times. The function itself only needs to be there once along with the loading of the data. The example also had an exit in it which included in your script would have stopped the page dead in its tracks.

I’ve reorganized things a bit, by placing the the call to get the data at the top, the function that gets the specific data at the bottom and the specific two calls for the min and max values in the table area.

I can’t really test it here because you are using some functions I don’t have available and my template structure is quite different now, but attached is a text version of the script.


csc0709-02.txt (14.8 KB)