PALGN really slow?

[NeoSanity]

[desmodontinae]

[Discosis]

It's 1am (actually by the time I finished writing this, more like 2am). The server is slow. What's a guy to do? Well, if you're me then you're checking out why it's slow.

What I've found is that when the server responds slowly it's generally because of locked processes in MySQL. At the time I investigated this particular bout of slowness I found ~60 queries locked in the database, some for over two minutes.

When queries lock, pages don't get generated, and the processes that are supposed to serve them hang around taking up memory.

Let's talk about memory for a bit. The current server has 2GB of RAM. This is not a lot for a server. A normal PHP process generally likes around 16 Mb of RAM. It doesn't use it, but it *can*, which is important. This means on a webserver with 2 GB of RAM you can theoretically serve 2048/16 processes - 128 simultaneous requests. This value is important, because *IF* every PHP process on the system *WERE* to suddenly consume all available memory for something, that's the highest number of processes you can support without paging to disk. As James pointed out, paging to disk is very expensive for a server, mainly because things run some order of magnitude slower than they should, which just slows EVERYTHING down.

By the way, PALGN's PHP processes are actually configured to use 128 Mb of RAM each, probably because at some point it became necessary to raise the limit to stop running out of RAM while serving a page.

Of course, that's a webserver. PALGN's server is a webserver, a DB server, a mail server, and a DNS server. There's a few other services in there, but I think you're getting the point. As James has previously pointed out, MySQL likes memory. It's a database, you'd expect it to do that.

(Interestingly enough, right now as I write this and the server's slow, we're NOT running out of memory. It's close, but we're within the limits of physical RAM on the server.)

So in this environment (where we really can't support a lot of simultaneous operations), it's in our interest to serve pages as fast as possible and clear that slot up for the next request to come along. This is very difficult to do when the database is slow.

So, why is the database slow? There's a couple of reasons. The first is simple - queries are being run that aren't hitting indexes. We need to find the slow queries and figure out what indexes to add to the tables to speed them up. It's amazing what effect this can have on a system. Secondly, it's because tables in the database are built using the MyISAM storage engine. MyISAM is nice and fast (which is one of the main reasons people use MySQL in the first place), but it doesn't handle exclusivity when writing data to a table very well. It'll lock the entire table when updating one row in it, so noone can read from or write to the table. This is called table-level locking, and it's pretty annoying. We can move to InnoDB, which has row-level locking (only the row being updated is locked, everyone's free to read from the table), but this is non-trivial, and InnoDB tends to use more resources -- if you'll recall, I just pointed out we're pretty short on RAM.

What do we do about it? Well, we get off this server and onto a newer one. This has been a point of contention for a while because of the age of some of the PHP code running on the site. There had been information going around that we're running PHP 4 and the code won't run under PHP 5. Actually, we're running PHP 5.2, and things are working. Thus it becomes a migration and server management issue.

The new server has more RAM, so we can run more processes concurrently. We fix the queries that aren't hitting indexes. We move to InnoDB. We look into caching things with Memcache to remove load from the DB entirely. We get the database working properly, and we can serve pages faster.

... for those of you who are wondering, that's what I do.

[Discosis]

By the way, I've seen the bills for running the current server, and for commissioning the new one. It's not cheap. If you'd like to foot the bill (there's 4 figures in it before you get to the decimal points) then feel free to contribute.

Otherwise, some of you could show a BIT of respect for the fact that someone's actually paying real money for this virtual playground.

[Discosis]

Also, no I'm not being paid.

[Cro]

*reaching for ban stick*
_________________

[Karai Pantsu]

Plenty of respect is shown here to those who have visibly put their time, effort, and money towards making this website and the forums worth being a part of. That is to say, the Staff.
_________________

[Cian]

[Smurf80]

[Fetidchimp]

This shit takes the fucking cake....
If this is how you fucks do business, good luck with that.

Then you can have all the ridiculous pop up ads you like because no one will be here to give a toss about it....fuck me, did you two go to the randolph and mortimer duke school of business or some shit, because last i checked, pissing off the people that you intend to try and make money out of, is not a good way to go about it.
_________________
kill, kill, kill.....the white man. Kill the white man, kill

[ThunderCurls]

[Benza]

this thread

_________________

[mikezilla2]

this benza


_________________

[Shnryu]

The site has become noticeably faster and the bank ad is gone. I'm only a random member nobody really knows but i like this site all the same, so if its because of the work Discosis is doing, then i just wanted to say thanks.

[Island_Wolf]

Discosis, not trying to be rude but James had already explained the current state of the server, issues with it etc many times in the past in a tad less technical way. He had blockers to do improvements (I feel sorry for annoying James all these years about the website). Upgrade is a good thing, the secretive isn't.

Being a developer, I know what is like to develop things in hostile environment and I never liked it so I do feel for you. I have web applications that no one used but had to develop it anyway, for me, this is just the case of people not using it. I know you are just doing your job but what's the point of developing/spending away if "no one" (PALGN community) is going to use it?

Since there's much hostility (apologies if that is not the right word), maybe halt the big changes to the website (just focus on migrating to the new server) and start focusing on listening to our needs? This is been dragging on for way too long now.
_________________
"Work hard now, play even harder later"
I am an Industry Ambassador at Girl Geek Coffees (Sydney)
Unofficial PALGNchat - IRC Client *New Server!*

[Fetidchimp]

[BurnZ]

[Jarrod]

_________________

[ThunderCurls]

Rollin' Rollin' Rollin'

Keep movin', movin', movin',
Though they're disapprovin',
Keep them doggies movin' Rawhide!
Don't try to understand 'em,
Just rope and throw and grab 'em,
Soon we'll be living high and wide.
Boy my heart's calculatin'
My true love will be waitin', be waiting at the end of my ride.

Move 'em on, head 'em up,
Head 'em up, move 'em out,
Move 'em on, head 'em out Rawhide!
Set 'em out, ride 'em in
Ride 'em in, let 'em out,
Cut 'em out, ride 'em in Rawhide.
_________________
Want cutting-edge Australian video game journalism with a friendly community? Click here!

[Benza]

I saw the rollin, rollin, rollin and immediatly started reading that to the tune of Limp Bizkits Rollin'....

... this is not a fact I'm proud of.
_________________

[fatpizza]

[theory]

_________________

[Cyph]

Wow, this thread has picked up. Love the popcorn image Benza.

In all seriousness.... how hard is it to make a new forum and for us to move to it?

[Fetidchimp]

probably not so hard for us, probably a bit harder for jeremy and co.
And really, they shouldn't have to.
_________________
kill, kill, kill.....the white man. Kill the white man, kill

[Kurupt]

I would just like to thank everyone for making this the biggest thread I have ever started on any forum.

You're all cool dudes. <3