I think I have a little understanding of MQTT but I need a help. Do I need an MQTT Broker? Or is WD handling that function? And how would I implement MQTT on my webhost side so I can continue to display my clientraw data? I don’t understand how I “switch” from using ftp transfer to using MQTT.
I’m guessing WD will send my clientraw data to the mqtt broker, and somehow that data is sent to my webhost…but it’s not clear to me how my webhost will receive that data…it almost sounds like I need a client that will run on my webhost server?
After reading another thread about the use of MQTT, I decided to search the Internet for some (beginner) information … except, after about half an hour of reading different websites, I was more confused on how to use MQTT… :roll:
What I would really appreciate would be a (fairly) well written MQTT tutorial - I really hate having to watch those video tutorial…
That would be nice. I believe I have most of this figured out but having trouble with how I get the clientraw files to my actual webserver. How I think it works is you first get a broker/server, there are free cloud services available like Eclipse or Mosquito, and once you have an account you setup WD to “publish” your data to that broker. I suspect there is a client I would need to add to my webserver to handle requesting a subscription to receive the data A.K.A. clinetraw data files from the broker. But at this point I am only guessing this is how it works.
My initial thoughts…it would be nice to get rid of ftp for transferring my data to my webserver. But so far it appears that setting up MQTT is not an easy solution, and, perhaps not even possible unless you run your own webserver. I don’t know if I would be able to continue using the Saratoga scripts to display my wx data, or if MQTT would require every visitor to “subscribe” to my data in order to view it.
So it seems there is a lot to learn in order to implement this solution…still totally confused
Im trying to use MQTT as well but I don’t get any data in graphs (use Grafana)
I did setup nodered (see att 1) you can see I have data in debug modus
You have to setup a database in influxdb ( i did but difficult to explaine see other posts about that)
I use grafana for graphs (see att 2)
you need to set it up as in att 3
I always seem to take a different tangent when new things come along…it appears that misunderstood what MQTT is about and how I could use it in my setup. It was fairly simple to setup a free broker and have my clientraw file sent via MQTT by WD. But then what? I thought it should be possible to use MQTT to publish my clientraw file to my webserver allowing me to get away with not needing FTP. Because my website uses templates from Saratoga and W34 it seemed logical that if I could get my clientraw file updating on the server I would be home free.
I should mention that I like both Saratoga and W34 scripts and I want to continue using them.
My quest now is to figure out how subscribe to my MQTT publication and save that data as clientraw.txt on my webserver.
Please, please, if and when you figure out how to use the MQTT data, pass that information here…
I also use highly modified versions of Ken’s Saratoga templates as well as the ‘Personal Weather Station Dashboard’ (ie: PWS_Dashboard or Weather34) scripts.
I am using the WiFiLogger in one of my console - the WiFiLogger is able to produce the MQTT data but it need a MQTT server to receive (link and specific port number) that data…
Since I operate my own weather web server (mostly for my personal use), I will have to install an MQTT broker/server on my server somehow… :roll:
The long pole in the tent appears to be my webserver host. I can’t figure out a way to use MQTT on my host. They lock down all ports, and I understand the need to do this for security of their system. Unless I am completely missing the jist of MQTT, I can’t fathom any way for a commercial host to allow these MQTT outbound connections from their servers. It seemed simple when I started…WD publishes my clientraw data…so all I needed to do was subscribe to my broker and I’d be all set. But therein lies the problem…how to get past the firewall on my webhost to allow me to subscribe to the data.
I have a basic understanding of MQTT and how it could be used to reduce our dependency on FTP connections and processes. The question is, how do we make our wx data presentable to the public via an webhosting server? I do not want to run my own server, and, I can’t see how any commercial hosting service would allow these outbound connections to a broker.
My mind keeps mulling this over and over, and the only possible solution I can come up with is to move the client subscription off the webserver and onto the subscribers machine.
Anyways, this has been an interesting quest for me to learn about MQTT. I would love to hear comments from others who have been down this road and how you implemented MQTT to display your wx data, and specifically, how you circumvented the commercial host port issue. Perhaps you are able to host a broker on your webserver? Or maybe you figured out a way to run a mySql database and import the subscription data directly?
Just thinking out loud…but if you run everything on your local lan you probably don’t use ftp…why would you need/want MQTT? The main reason I think people use MQTT is that they can publish data to one source, and in theory, have thousands of subscribers to that data.
Like I say, I probably have a lot to learn about this MQTT stuff.
Another way might be websockets to
Get the clientraw data to your web page
I would need to build a program to do that
But it would also be needed to be installed and
Run on your server
Every file needed for the weather web pages are uploaded (FTP) from Weather display to the server specific directories/folders where they are needed to produce the web pages.
I would like to learn about the reliability and the use of MQTT - so, I would like to have an MQTT broker/server/client on my home office server.
And so do I… :oops:
How about we learn together…
Maybe, just maybe, as we go along the ‘learning’ of the MQTT we could also put some notes down as a possible tutorial for others to learn from and to use… Indeed, at 72, I NEED to put notes down otherwise the limited amount of RAM left in my brain has a definite tendency to forget things, the short term memory is the worst… 8)
Just to note the thread on the MQTT Dashboard (http://discourse.weather-watch.com/t/67293) may help - i have also posted a Grafana via influx if anyone is interested - once its running then MQTT is a better option than FTP. Although i do note the MQTT cron does not seem to reconnect if a connection is lost, i may post that as a possible bug (?) as it does drop the reliability a little at the moment.
Andy, we (NorCal Dan and I) have been following both threads … I think. :?
It is my understanding that NorCal Dan might have created this particular ‘current’ thread so that the basics on using MQTT are explained … such as a beginner tutorial on understanding MQTT, on how to setup and use MQTT to upload the data produce by our weather software (ie: Weather Display).
I consider the thread you have started more advanced, such for applying already gained knowledge of MQTT to create a dashboard - even after reading all the messages of that thread of yours, I still do not understand the basic of MQTT… :oops:
Have you found a ‘known’ way for MQTT to reconnect or is that one of the weakness of MQTT - just curious?
I’ve read this thread and found that it is far too advanced from where I am currently. As I mentioned in my earlier posts, I would like to continue using the Saratoga template for my website. Your thread Andy appears to me to using custom webpages written to work with your MQTT data.
Since I have no experience using MQTT I wanted to start at the beginning and make an attempt to use MQTT instead of FTP. Initially it seemed like a smart move to create an easy way to get my wx data to my webhost without all the overhead and security issues associated with FTP.
I have WD publishing my clientraw.txt data to broker.hivemq.com and it works quickly and dependably. I could have setup my own broker like mosquito on one of my local machines, but I figured with all the free brokers available why not use them.
Now that I have my clientraw data being published, the next step was to get that data to my webserver. Here’s where I ran into a problem. My webhost, fastwebhost.com, locks down all ports except what is needed to serve webpages and ftp. MQTT has assigned ports 1883 and 8883. Initially I was able to use the broker.hivemq.com websocket port 8000 from my webhost to subscribe to my broker, but they apparently discovered I was using this port and locked it down.
I have some ideas I want to try to work around this issue with my webhost and will continue to update this thread if I make any headway.
Now, even if I manage to get this problem solved I still have one more issue that needs to be resolved…how to convert the subscription to my clientraw data into an actual clientraw.txt file on my webserver. I suspect I will need to either parse that data separate fields, or, use the custom option from WD to create separate tags to import into a database on my webhost and then use that to update my wx data.
