Musing on memory useage

Page faulting isn’t done by the program itself. It’s a feature of the operating system (I’m sure you know that anyway). So WDWatch isn’t actually causing the page fault…it’s asking for memory to be allocated or accessing memory that has previously been allocated but has been swapped out onto disk. Assuming the program hasn’t gone crazy and is continually asking for loads of memory and then giving it back, then it suggests that the operating system is swapping some of the program or the program data area out to disk for some reason. The most common cause is lack of memory.

How much real free RAM do you have (be careful not to look at a value that includes free paged RAM as well as free real RAM)?

Chris my current free physical memory is 163mb out of 384mb according to the task manager. As I said earlier I am not very familiar with Windows memory management but I have always assumed it should only page out when there is not enough physical memory to satisfy any particular request and then it should page out on a least recently used algorithm so, in theory at least, heavily used pages are the last to go out.

This machine does run a few tasks other than WD and MySQL, since it was new I decided to run a couple of monitors to check on temps etc. Now I know the machine runs reptty well I will try turning off some of these and anything else which looks like it is not really essential to see what if any difference it makes. After all 384mb of memory is quite enough to run many applications on most PCs so at the moment I am not inclined to give it any more until I am absolutely sure it is the only way to go.

Stuart

I have noticed no difference overnight having shutdown some monitors etc which are not needed.

However what I have noticed is that there must be a memory leak since my WD memory has gone up now to 81,916k from the 70,800k just over 24 hours ago. So I guess it is time to start to disable some WD features. I dont use many, I have no webcam, only 2 custom screens, dont use the standard web pages and I use customise file creation to upload. I do use MySQL and WDL so I am creating clientraw.txt. Handles by the way have not moved from 191 since yesterday

Probably the only thing I can turn off for a while and not screw up my web pages too much is the custom screen, which if I had to bet on what was causing the leak would be the function, bearing in mind other issues.

Stuart

IME the memory used by WD will creep up as time ticks by, but using the 3 AM minimize option will reset it daily and the system will be stable for long periods.

I actually leave it minimised all the time as I nearly always use the viewer on my other m/c and these are the storage measurements while minimised!

Stuart

the real time graphs in wd, as more data is plotted, do use more memory
until they reach the reset point and then they should clear and start again
you can set that reset point lower , see in the real time graph setup, and or even clear each real time graph and check the memory use of weatherd.exe and see if it drops

I’m not sure that has the same effect.

I ticked the minimise at 3am anyway but unfortuneately WD crashed/hung before it got to 3am… so I will have to wait and see what if any effect it has on the memory useage.

Stuart

I don’t think it’s the same. I’ve WD minimised (nearly) all the time and (with 10.27d) I’ve more free memory left when I wake up than before I turned in …

wedensday here, WD 10.27d still going OK for me (since sunday)

we are in wellington, met up with Leo last night :slight_smile:

wow that minimise at 3am does have a massive impact on memory usage!! I tried it here last night for the first time and the reduction was from about 80meg to less than 20! :slight_smile: of course its climbing as the day goes on but it sure is a good cleanout

Yep, that was my experience :smiley: once I had discovered that check box it would just about run forever.

I’m running a machine using Win98, so I ticked that “Free memory at 10pm” and that is also working really well in managing some crashes I was starting to get at regular intervals

Well now WD has survived far enough to get to 3am it sure does drop off memory useage significantly even when WD runs normally minimised anyway. Brian must have some extra code in there to do this I think otherwise WD would free up memory when you minimised it, either that or there is some kind of special hidden minimise function in Windows!

Because of my background on mainframes I still am not happy about the huge amount of paging which seems to go on and I’m going to take a look with some new tools I’ve found to see if I can find anything out as to why, especially with wdwatch. That amount of paging HAS to have an impact on performance since every page fault has to be at least one I/O read.

Also maybe I am being confused by windows terminology with memory. The task manager shows memory use in the grap at about 314252K but in the box below the graph it says 202000K approx (it changes a bit) available. Now I was assuming available meant free but I think in this context it probably means that it could be free OR could be used by another task by being paged out. And yes the 202000k probably includes the actually free (unallocated) memory as well.

Stuart

All the different types of memory are rather confusing! I’m not even going to try to guess what you’re seeing.

One thought about page faults is that I (we?) tend to think of them as paging memory to/from disk. I’m not sure that is necessarily the case though. I think a page fault is just any memory context switch. It could be between memory and disk (or disk and memory) but it’s possible that depending on the memory model used by the software it could also be a memory to memory switch.

It’s a while since I’ve done any C/C++ coding (and I don’t know if Delphi works in the same way) but there are various memory models that you can compile for, e.g. Tiny, Compact, Huge, etc. These are all related back to the legacy Intel memory models, where in the 16-bit world, you could only address 64k of memory at a time. So if you wanted to use more memory you put different parts of your code or data into separate 64k (or smaller) memory pages. If your code was running in page 1 and you then wanted to run a routine in page 2, the operating system/software library code triggered a memory context switch to swap code pages. These may appear as a page fault.

Having said that, 32-bit software usually works on the Huge model (4GB addressable space?), so you should only get page faults when you swap between real RAM memory and virtual disk memory (unless you can afford 4GB of RAM in your PC!) However, I’m wondering if Delphi still uses memory paging, or if one of the libraries Brian has used in WD is really a 16-bit library rather than a 32-bit one? If so, then maybe there is memory to memory paging going on as well (using the wonderfully named twunking mechanism). Memory to memory swaps aren’t as big a problem as disk/memory swaps…they’re only changing address registers pointers…but they can slow things down which is one reason why 32-bit computing is faster than 16-bit computing.

Stuart, I’ll be interested to see your expert conclusions. I don’t know if different mainframe OS’s are designed differently, but something to bear in mind is that (I believe) Windows memory management from NT onwards was devised by the same folks that wrote VMS, my favorite OS but also one that that seems to love pagefaulting.

Different O/S’s do have different memory strategies. VMS->WNT are the letters incremented by one for a good reason? Well yes, some of the VMS team did indeed move on to MS to work on NT. There are some similarities between WNT and VMS and the page faulting mechanism is one of those. VMS was developed back in the days when memory was expensive, so it’s big claim to fame was that you could pack more programs into the same machine because it virtualised the memory…as long as you didn’t get carried away and cause disk thrashing (and the stories about disk units walking across the floor when thrashed hard!)

We’ve still got VAXes active at work running VMS and I still log on from time to time, but haven’t done any development on one for about 11-12 years.

You know what an expert is … and ex is a has been and a spurt is a drip under pressure… :oops: :oops:

Seriously I am no expert in MS systems. So far I have not found anything to put my finger on. It is just that a program like WDWatch to me seems strange for it to run up so many million page faults when all it is doing is checking that WD is still alive…

Stuart

I was hoping your research would be better than mine :lol: I’ve found many articles about how win memory management works and how to improve it by fiddling with the settings, and just as many articles debunking the same theories #-o

Having done a bit more on this I now find that my WD system is running at some 650 page faults per second and it has over 200mb available memory. However this tells far from the whole story.

Firstly let me say what I have found applies to W2K and probably XP, W98 and WME run very different memory models.

There are two main types of page fault, there is the ‘hard’ page fault where the data has to be read in from the page file on disk, and there is the ‘transition’ page fault. Now the transition list is where a page marked for page out to disk is put while it is waiting for the I/O to complete, the page is marked invalid in the page table but has not yet been re-allocated so the data is still as it was. When the memory manager gets a page fault it checks to see if the page is on the transition list, if it is no I/O is needed the page is then marked valid and the faulted code can be resumed. So a transition page fault is very quick to be resolved as it only takes a few cpu cycles and no I/O.

My WD system runs with 450 tansition faults per second, so if I read the info correctly I am only getting some 200 ‘hard’ page faults a second. This is not so bad but still bad enough in my view.

The available memory can lead people to think that this equates to free memory. I believe this is not the case, available memory is the total of any free (unallocated) memory AND any memory which is allocated but is also pageable and therefore available for other applications to ‘steal’. So looking at available memory does not tell you how much unallocated memory you have. It would seem to me that in a windows model you need an awful lot of memory to reduce paging (both hard and transition). It would appear that the W2K task manager memory graph gives a better idea of memory allocation, on my WD system it sits at 313604K (this is also shown under the Commit Charge) which means that since I only have 384mb available physical memory it would seem that the maximum unallocated memory I have is about 71Mb an awful lot less that the 247000K shown as available. You dont need very much to go awry and all of the physical memory would be eaten up and paging would then take off even more.

Even adding shed loads of RAM will not prevent paging, my m/c I use for most of my PC work has 1Gb of physical memory and even with not a whole lot running it still pages stuff in and out (altough not nearly as much as the WD m/c pages) and it currently has over 600MB available and the Commit Charge sits at 539MB roughly and so most of this available 600MB is actually unallocated!

So to sum up… hmmm… Nothing conclusive I guess except to say what most people have always said about Windows, add as much RAM as you can when building a system. If you have a slowish processor the more RAM the better.

My WD machine has only two ram slots with currently a 256mb and a 128mb in each, I will probably replace the 128mb with a 512mb, giving 768mb in total. This I hope will reduce the paging somewhat and if memory useage does peek above the 384mb then it will certainly help.

Having read quite a lot about windows memory management it does appear that there is memory which can be allocated which is not pageable. I am wondering if Brian could take a look at the memory used for data which is shared between WD and say WDWatch to see if anything could be changed to ensure the shared data stays in memory, this might help with the huge numbers of page faults which WDWatch experiences?

Stuart