Tweaking the firewall again. It’s difficult checking if the changes are working because the ‘guests’ seem to arrive in waves. So I make a change and think it’s better only to find they’re having a collective coffee break.
I think the firewall is handling the ‘guests’ more stably now. I’m seeing 40-50 dropped packets per second from the ‘guest’ IP address ranges. If every one of those were hitting the forum and requiring a response that would definitely slow things down.
I’m now convinced that the ‘guests’ are causing the recent problems. I’ve been working on the server for about 3 hours and whatever I’m doing I can tell when the next blast of them arrives without the firewall handling them properly. I can even feel the slow down in my SSH sessions so it’s not just the forum/web-server that are affected.
Seems to be loading up nice and fast now. 173 Guests. 6 Users. Good job on server/firewall.
120,000 attempts to connect rejected since I last restarted the firewall (perhaps 6-8 hours ago) so that gives some idea of the impact the ‘guests’ were having. They’ve slowed down their attempts from 40-50 per second to about 5 per second now, possibly because they see they’re getting nowhere.
I’ve also added some new spider definitions to the list the forum knows about so you can see how many of them are hitting the forum (in the online user list at the bottom of the forum homepage). The online guests is much lower now that many of them have been identified as crawlers/bots.
good work Chris! 
A good Job Chris,
tho’ it’s rather nasty having to identify everything and manually change the firewall rules…
Appreciate all your work on keeping the forum running and clean. (and to your admins also).
Ken
Ahrefs (101) - Can we block these guys since their spider is out of cojtrol?
I want to do a bit more analysis on that first. It looks like Ahrefs scan from about 100 different IP addresses where Bing scan from about 4. However, the number of accesses from Ahrefs is about 10 times lower than Bing so they’re doing less scanning just from more addresses.
Interesting 8O
All this is very interesting but does not yet explain my issue. No matter whether I am on Linux or Android the initial access to the forum is very slow, but once on it behaves normally even if I go back after viewing othe sites. If I close down the system and later in the day visit the forum again the initial access is again very slow. So from my perspective the issue seems to be validating my user on the system. Any ideas why? None of this has changed significantly since my first report. What if anything could this be my end?
Stuart
The issue does not have to do with validation of user. Try an incognito tab and you’ll notice that even when not logging in the problem is there.
How slow is “very slow”?
I’ll time it next time, but I remember it taking well over 30 seconds from clicking on the bookmark before and the browser in the message area saying waiting for weather-watch.com so it was past the DNS bit.
Stuart
I’m looking at your logs now Stuart and I can see 30-50 second delays between your browser asking for the index page and then asking for the rest of the page components after you come back online after a while offline. Unfortunately there’s nothing in the web server logs to say why there’s a long delay. It could be that it took the server a long time to send the index page or it could be that your browser took a long time to process the index page and ask for the other page components, or it could be a bit of both.
The only thought I have at the moment is…does your browser auto-clean the cache after a while or are you restarting the browser for each session and the cache is being auto-wiped when you shut it down? First time in you have to reload the cache so it’s slow, but looking at the second page there are a lot of page components cached so you don’t need to grab them again. An hour or so later your cache has been wiped/or you’ve restarted the browser and the cache has been cleared and you have to start over?
My only other thought is can you run Fiddler (or similar) to see if you can see what’s happening from your point of view?
I’ve blocked the Ahrefs Bot. It’s crawling web sites to sell the data to SEO marketing people so it’s pretty unlikely to be of any use to weather-watch.com. It may take a while for it to read the robots.txt file and find it’s blocked but it claims it will do that at the start of the next scan. If this doesn’t work I’ll block the IP addresses next.
why log out?
why not just stay logged in?
then no delay
Even in an incognito chrome session I’m not seeing delays like that, maybe a good second to get the index page up, and a second or less from hitting the login button. I don’t have a screaming fast internet connection like most UK users either…
Good move Cynical old me wll be quite surprised if they obey robots.txt…
Chris I have tried Waterfox and Chromium on Linux, Waterfox on Windows 7 and Firefox on Android and they all suffer the same initial problem, again this morning just over 30 seconds on Waterfox for Linux. Since they all have their own cache handling I’m not sure what that means. I’ll look to see what I can run to trace this.
Stuart
Running HTTPFox to trace what is happening the delay seems to be to a site called ocsp.digicert.com, once there is a reply from this site the rest of the data flows quickly and the page appears.
(Request-Line) POST / HTTP/1.1
Host ocsp.digicert.com
User-Agent Mozilla/5.0 (X11; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0 Waterfox/56.2.13
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-US,en;q=0.5
Accept-Encoding gzip, deflate
Content-Length 83
Content-Type application/ocsp-request
Connection keep-alive
is the request header and
(Status-Line) HTTP/1.1 200 OK
Accept-Ranges bytes
Cache-Control max-age=103060
Content-Type application/ocsp-response
Date Wed, 04 Sep 2019 08:10:37 GMT
Etag "5d6e5f37-1d7"
Expires Thu, 05 Sep 2019 12:48:17 GMT
Last-Modified Tue, 03 Sep 2019 12:40:23 GMT
Server ECS (lcy/1D1C)
X-Cache HIT
Content-Length 471
is the response header.
Subsequent accesses to the forum (without any delay) does not show this request
Dont know if this indicates anything or where the request comes from.
Stuart