MYSQL Data Import

Archive Archived Weather Display Problems
1

Having used the access dbase for over a year now and seeing things going a bit slow (900K rows) I thought it was time to move onto a proper dbase.

I’ve installed MYSQL, created a dbase , run the script to create the table and got WD to see the dbase. all ok an working well.

I’ve tried to import one months data, so far its run for 4 hours and has imported 16k rows, from memory each month has 40k rows of info, Task Manager shows mysql & wd using 95% of the processor,

Is there something I have done wrong or is this normal ? I just used the new tool in WD to import data.

Thanks

Keith

2

hi
which version of WD?
please try the latest version, or at least 10.15p,as it should be much faster…
let me know

3

Thats the one (10.15p)

I killed the process after 8 hours as it had only done 20K records. I checked the records and everything looked ok its just very slow.

Cheers

Keith

4

hi
i cant do much about that unfortunately
but the normal 1 minute logging will keep going
so maybe just leave it going and dont worry about it?
but, though, you may have to reset the graph history files, for the current month, via action, convert wd log files to graphs…

5

Thanks , I think I’ll do as you suggest.

Is there a way a SQL script could be created that would do the import ?

Cheers

Keith

6

the slow seems to be with the append command (i tried insert too)
if data is added to the same row, its very fast
but it the append process that slows it down (i did some experimenting)

7

I know something about Oracle but not SQl. In Oracle you could run a batch file that only did the commit at the end, does MYSQl work this way too ?

8

One thing I found about mysql is that it does not run fast if there are any other processes running, even low priority processes such as seti@home. One of the queries I use takes 6 seconds without seti as a background task, but 38 seconds if seti is running. Mysql should preempt Seti@home, but it does not, it just seems to sleep a lot if the cpu load is high for any reason.

/Ian

9

Can I suggest that the Delete All Records has a health warning added to it.
Its quite a powerful button,and can create an awful lot of work :!:

Also I tried running the import again, its just as slow but watching task manager all the work is being done by WD. Its also using a lot of memory I peaked at 67meg

How about an export to a csv file and then an import script to Mysql

cheers

keith