Forum speed issues

[Trinitrotoluene]

I was advised to post this here. It seems that the forums tend to timeout a lot, I've seen queries take 20-30 seconds to be returned sometimes, and other times posts fall into the abyss because submitting them times out. From my experience with SQL it almost feels like they're is to much load on your SQL server - just thought you guys should know!

Edited July 31, 2006 by Gavsto

[Saru]

Do you find that this happens more at any certain time of day ?

[Trinitrotoluene]

To be honest no, it feels sporadic and random which leads me to believe the problem is on your DB. I get the same problems on my work machine, my home machine and at my girlfriends house. Are you aware of the effect I'm describing? You click and wait, and it will wait and wait, and when it's bad take up to 30 seconds to return a page. But then you click the next second and everything is fine again. I've no idea what your server setup is, I will assume IPB through mysql db driver? The only time I have seen this happen before is when SQL reaches it's 'limit' and stops processing new queries until the backlog is resolved, this is of course a guess and I'm probably wrong

Gav

[GreyWeather]

does with me, but I'm just guessing the bandwidth is being over-run by to many people using the site. normally 7-8pm, or 10-11pm in GMT

[Saru]

I get exactly what you describe, but I've assumed it was some sort of routing issue with my ISP because there never seems to be a corresponding server load or page generation time increase, and when it happens I can be talking to someone else who says they can access the site no bother at all, which wouldn't make sense if it was a server problem.

Up until now, nobody else has reported having the same problem.

Are you in the UK ? Your ISP isn't BT by any chance is it ?

[Trinitrotoluene]

No, I'm on blueyonder/telewest and I've never had any problems (other than on this site). What I can definitely tell you is it is a server side problem and not client side. If you look in the firefox/ie status bar, it will tell you it's actively waiting for a server side response. There's definitely a page generation increase, you will probably know all ready but it tells you at the bottom of the thread. This page was generated in 0.1778 seconds this time, but if I were to click submit and it was to do it's normal lag, instead of stopping and trying again, let it go through and then look at the figure. I've seen it hit 35 seconds before! I think because it seems so much at random, it's impossible to get other people to verify, because by time you've asked them the problem would probably have cleared itself.

Edited July 31, 2006 by Gavsto

[Pax Unum]

I was advised to post this here. It seems that the forums tend to timeout a lot, I've seen queries take 20-30 seconds to be returned sometimes, and other times posts fall into the abyss because submitting them times out. From my experience with SQL it almost feels like they're is to much load on your SQL server - just thought you guys should know!

I have this happen often... and it happens when its busy or slow, maybe a little more when it's busy, not sure though... I've lost several posts, I save a copy to clipboard now before I 'Add Reply'

[Saru]

Ok well thanks for bringing this to my attention, i shall be looking into this to see if something can be done about it.

[Trinitrotoluene]

Saruman,

A few suggestions for you: If you have direct access to your SQL server then you can run either Performance monitor (start - run - perfmon) or SQL profiler, if you use perfmon look for something called SQL Server: Latches, you can monitor Performance bottlenecks based on the number and length of waits for internal resource locks to be granted also SQL Server: SQL Statistics which monitors Query compilation speed and overall efficiency of the query optimizer. Minimizing compilation time and re-compilation frequency improves performance. I can go into more detail if you need but I'm sure you have a tech guy hidden around somewhere!

Gav

[Saru]

Unfortunately I don't have direct physical access to our servers, both our linux boxes are in a datacentre across the Atlantic.

So far it looks like another memory upgrade is in order for our mySQL server box, there's a bottleneck with the server waiting for hard disk access as the available memory has been used up and we're onto the swap file. Getting large spikes of IOWait which seem to correspond to the delays in viewing the page.

I'll check everything out thouroughly to make sure there's nothing else that might be causing the problem. If not, we can usually get a memory upgrade performed within a few days.

[Trinitrotoluene]

Well I guess that explains it! Glad you found the problem (not so glad I bought to your attention something that costs money to fix lol )

[coldethyl]

I am in the States and I also experience this problem. Just letting you know.

[Saru]

I've made some changes to the database server configuration, is anyone still experiencing this problem ? It seems to be working a lot better now.

[Trinitrotoluene]

Seems to be working fine, I'll report back here If I notice the same lag!

[Trinitrotoluene]

Still lagging, and before it lagged for one of my mates at exactly the same time.

[Waspie_Dwarf]

I concur with Gavsto, there are still problems. Until this thread I had assumed that the problems were my end (I'm not very computer savvy and wouldn't know the difference between an SQL and a dyslexic squirrel).

I've just made a post that took 6 minutes. The first attempt hung until I got a "page not found error". The second attempt hung for a couple of minutes before being accepted. This was at almost exactly the same time as Gavsto ended up with a triple post in the same thread.

[Saru]

Ok, what i've done now is changed the way the forum search looks through the database when someone enters a query in order to improve efficiency and lower resource usage.

When this problem was first reported, I set up a special log file for recording all the database queries which took a long time to complete, and most of them have turned out to be forum search queries.

This change should not only lower the resources required to conduct a search, but also drastically improve upon forum search times.

If you still experience the same problem again now, let me know.

[+Link of Hyrule]

Hmm, is the latest work you did the reason I had a "site off-line" message when i was on earlier tonight?

[Saru]

Yes I had to take the forum off-line for about 10-15 minutes.

[coldethyl]

I had a post take about 90 seconds. But that was only one for a really long period of time as opposed to it happening several times for about 30 seconds a time.

Am I making sense??

Edited to add that it happened at 1:40 pst.

Edited August 3, 2006 by coldethyl

[Daughter of the Nine Moons]

Hi SaRu

Just a heads up. For about the last 5-10 minutes I was having problems accessing UM. (other sites were ok) It started with searches that wouldn't conclude and then I couldn't access the forum main page. There were only about 500+ users on at the time as well.

[Saru]

That was caused by Apache being restarted, not sure at the moment what caused the restart. Looking into it.

[Phenomenon]

There were only about 500+ users on at the time as well.

I've just a serious hang that lasted for about 3 or 4 minutes. I can't say there's a time when it happens, no specific time of day when slow loading is an issue, but one thing I did notice was the guests online.

On one ocassion I checked the online users, of one entire page only 3 were actually active users, one was a guest and two were members. As for the rest, I spoke to invision a few months back because I had a similar issue. They weren't too helpful, but the server engineer at my hosts was.

That kind of activity, which I assume is bot related does use resources yet as far as someone being on the site is concerned, they are not an actual visitor in the real sense of the word.

My problem has been rectified, although I'm still not too clued up as to how.

Excuse the quality of screenshot:

[attachmentid=27380]

As you can see, there are only 3 actual visitors to the site, the rest is bot/engine related. I think if you can manage this then the problem with resources may ease.

[Saru]

We've always had a lot of bot activity on the site, it's never been linked to performance problems. Bots/spiders count as 'guests' and therefore don't have access to any of the site's most resource demanding features, and their contribution to server load is most likely to be very small. From the slow query log, the database queries which were creating the delays were all from resource intensive member only features such as the forum search.

It's also not a good idea to disallow search engine spiders from accessing the site, as they are a positive presence. Some search engines register as multiple 'users' coming from different IP addresses in order to spider the site faster. Intentionally blocking them would only hamper the update of our pages being indexed in the search engines and the generation of our contextual adverts.

Regarding the issue covered in this thread, slowdowns can occur at the best of times, for many different reasons. It still looks as though a memory upgrade would be welcome on our mySQL box, but the changes i've made to the search feature do seem to have significantly reduced the occurance of page timeouts.

I'm still working on it.

[+Link of Hyrule]

Has UM been having speed or connection issues recently? About 4 hours ago I couldn't get on the site at all, and now speed's really affected when I post a reply.

Thanks, PA

Edited August 20, 2006 by Paranoid Android