Name: none none

Level: Administrator

Re: The Sims Freeplay Preview (1 year ago)

test

Server transition (1 year ago)

Hi folks,

We're now running on the new server. Given the bulk of this code is from 2004 this is no mean feat

Current known issues:
- Nonprintable (?-in-a-diamond) characters are appearing in a few places, but this doesn't appear to be breaking any functionality.

Fixed issues:
- palg.nu (the URL shortener) had some issues but is working again

(I'll update this list as or if new issues come to light)

Please email technical@palgn.com.au if you notice anything screwy.

Jason

Re: Downtime, server move (Saturday night) (1 year ago)

Test.

Downtime, server move (Saturday night) (1 year ago)

Hi guys,

There'll be some downtime tonight while the database is moved across to the new server and crons / processes / background tasks / etc are brought up.

This will probably occur sometime between 6 and 8pm and last at most about 2-3 hours (barring a complete and total disaster). A test run last week took about an hour, but it's best to be cautious.

It's Saturday night, so while things are down I suggest a visit to the local pub.

Jason

PALGN is moving servers soon (1 year ago)

Hi guys,

Just a quick note to let you know we're aiming to move PALGN to its new (faster, nicer, and generally better-set-up) server over the next day / day-and-a-half.

There may be a few intermittent slowdowns as we export data from one server to another - seems the disks can only spin SO fast - but that's probably normal given the current server

Eventually I'll take the site offline for a little spell during which the live DB will be transferred and DNS changed. I'd like to do the switch over this evening, but there's a large amount of data to be copied so it might not end up happening until tomorrow. I'll post another update when I know more.

There's a fair few moving parts involved in the transition - if you notice stuff busted post-move, please email technical (at) palgn.com.au to let us know.

Jason

Some PALGN tech updates (1 year ago)

Hi everyone,

I thought I'd post a brief note regarding PALGN's servers.

I've made some changes over the past month that I think have had a positive effect on page load times (indeed, the good old "PALGN really slow" thread has finally fallen off Page 1 of the General forum).

For the techies, some details:

Some basic stuff was changed mid-January:
- RSS feeds are now generated offline every 5 minutes
- Whos Online is now updated every 2 minutes

I also added some performance tracking at the time to allow me to log slow page loads and DB queries. On January 28 I made some further configuration changes designed to stop search bots indexing some pages that ran particularly heavy and slow queries on the site, locking a bunch of DB tables. Amusingly, the heavy queries centered around the +/- voting system that applies to each message ... every vote you cast kills the server! (Ok thats a bit dramatic)

Anyway, hopefully you've seen some improvement in page load times.

Here's some averages based on my logs:
Jan-23: 12.09
Jan-24: 6.79
Jan-25: 4.82
Jan-26: 2.93
Jan-27: 3.73
Jan-28: 1.81
Jan-29: 0.61
Jan-30: 0.42
Jan-31: 0.50
Feb-1: 0.37
Feb-2: 0.39
Feb-3: 0.61
Feb-4: 0.44
(you get the idea)

As the page generation times fell, we were also able to serve more pages to more browsers - on Feb 10 the site served twice as many hits as January 23. (Looks like something big happened on Feb 10, right?)

While these numbers are encouraging, we're not out of the woods - the changes were a bandaid, this is still an old and mostly unmaintained codebase and there's still the odd long-loading page.

I'm looking to move PALGN to the new server in the next week. There's been a litany of delays on this owing to things like domain name ownership, DNS delegation, Christmas holidays, health, and I've been seeking some clarifications regarding important services. But we're finally ready.

I'll be aiming to do this with a minimum of downtime but some will be unavoidable -- I'll post further updates regarding the server transition when I have firmer details.

Jason

Edit - My grammar skills are poor at 1am

An open letter to all PALGN users (1 year ago)

This is an open letter to all members, users, moderators and staff of PALGN - past and/or present.

It's time to stop.

What exactly am I asking you to stop? There's a number of things.

I'd like you to stop banning users and hurling such utterly hateful and disgusting abuse at anyone who simply disagrees with you. Stop editing users' posts to twist their words. Stop taking shots at the departing staff for airing their grievances. Just stop.

All of these things accomplish nothing. I'm amazed that you even bother. It's childish and immature, and it has to stop. No-one on either side deserves the kind of mud that's been slung around here recently. Nothing good can come of continuing this behaviour.

Please, for the sake of everyone involved - stop. Take a breath. Look around. Is it worth it?

Egos are bruised, feelings are hurt, tensions are high and offence has been both given and taken, whether intentionally or not. This is painfully obvious to us all.

It's time to start moving forward.

Many of you have chosen to leave PALGN. This is fine. No-one on this side of the fence intends to give you any grief over your decisions. I'll fight that tooth and nail.

It's time to start bringing a dialogue thats respectful of others and their opinions. Former staff are all still welcome to participate in the forums, should you want to, though I'm under no delusions that most of you don't want this. If you want to criticise the staff that have moved on for doing so, perhaps you should discuss the matter with them in PMs or email.

It's time to start acting like the professionals you want to be seen as.

You might like to think that no-one's watching, but in the last week you've got the attention of both journalists in the Australian gaming media, and publishers with whom you'll want to continue dealing with professionally for years to come.

This is the industry you want to make a career in. It's your passion. Many of you are extremely talented writers. Is this really how you want to present yourselves to your peers? How do you think this looks?

To the staff who are departing, I'm asking you all to please go on, and make an absolute kickass site. PALGN stands as proof that you're capable of producing awesome content. Isn't it time to start focusing your energy on creating, instead of destroying?

I wish each and every one of you well. Make no mistake I'm sad to see you go, but I wish you well.

It's time to stop this madness and start moving on.

Sincerely,

Jason

Re: PALGN really slow? (1 year ago)

Also, no I'm not being paid.

Re: Readers Questions for PALGN Staff (1 year ago)

I saw someone mention those earlier (complete with a screengrab). I can't say I've ever seen them, but that might be more to do with the amount of time I spend here.

I'll have a look and see what I can find.

Re: PALGN really slow? (1 year ago)

By the way, I've seen the bills for running the current server, and for commissioning the new one. It's not cheap. If you'd like to foot the bill (there's 4 figures in it before you get to the decimal points) then feel free to contribute.

Otherwise, some of you could show a BIT of respect for the fact that someone's actually paying real money for this virtual playground.

Re: PALGN really slow? (1 year ago)

It's 1am (actually by the time I finished writing this, more like 2am). The server is slow. What's a guy to do? Well, if you're me then you're checking out why it's slow.

What I've found is that when the server responds slowly it's generally because of locked processes in MySQL. At the time I investigated this particular bout of slowness I found ~60 queries locked in the database, some for over two minutes.

When queries lock, pages don't get generated, and the processes that are supposed to serve them hang around taking up memory.

Let's talk about memory for a bit. The current server has 2GB of RAM. This is not a lot for a server. A normal PHP process generally likes around 16 Mb of RAM. It doesn't use it, but it *can*, which is important. This means on a webserver with 2 GB of RAM you can theoretically serve 2048/16 processes - 128 simultaneous requests. This value is important, because *IF* every PHP process on the system *WERE* to suddenly consume all available memory for something, that's the highest number of processes you can support without paging to disk. As James pointed out, paging to disk is very expensive for a server, mainly because things run some order of magnitude slower than they should, which just slows EVERYTHING down.

By the way, PALGN's PHP processes are actually configured to use 128 Mb of RAM each, probably because at some point it became necessary to raise the limit to stop running out of RAM while serving a page.

Of course, that's a webserver. PALGN's server is a webserver, a DB server, a mail server, and a DNS server. There's a few other services in there, but I think you're getting the point. As James has previously pointed out, MySQL likes memory. It's a database, you'd expect it to do that.

(Interestingly enough, right now as I write this and the server's slow, we're NOT running out of memory. It's close, but we're within the limits of physical RAM on the server.)

So in this environment (where we really can't support a lot of simultaneous operations), it's in our interest to serve pages as fast as possible and clear that slot up for the next request to come along. This is very difficult to do when the database is slow.

So, why is the database slow? There's a couple of reasons. The first is simple - queries are being run that aren't hitting indexes. We need to find the slow queries and figure out what indexes to add to the tables to speed them up. It's amazing what effect this can have on a system. Secondly, it's because tables in the database are built using the MyISAM storage engine. MyISAM is nice and fast (which is one of the main reasons people use MySQL in the first place), but it doesn't handle exclusivity when writing data to a table very well. It'll lock the entire table when updating one row in it, so noone can read from or write to the table. This is called table-level locking, and it's pretty annoying. We can move to InnoDB, which has row-level locking (only the row being updated is locked, everyone's free to read from the table), but this is non-trivial, and InnoDB tends to use more resources -- if you'll recall, I just pointed out we're pretty short on RAM.

What do we do about it? Well, we get off this server and onto a newer one. This has been a point of contention for a while because of the age of some of the PHP code running on the site. There had been information going around that we're running PHP 4 and the code won't run under PHP 5. Actually, we're running PHP 5.2, and things are working. Thus it becomes a migration and server management issue.

The new server has more RAM, so we can run more processes concurrently. We fix the queries that aren't hitting indexes. We move to InnoDB. We look into caching things with Memcache to remove load from the DB entirely. We get the database working properly, and we can serve pages faster.

... for those of you who are wondering, that's what I do.

Re: PALGN really slow? (1 year ago)

We're working on it.

Re: Readers Questions for PALGN Staff (1 year ago)

Actually, there are a few differences in the conclusions, but they're not worth the public fallout and finger-pointing. RAM's an issue, but the DB design is a major problem. If your database gets slow, your site gets slow. When your site's slow, requests back up, if requests back up then the server drowns in incoming requests, runs out of RAM and pages to disk. And we're back where we started. RAM's an issue, but we were unable to add more to the old server.

There's more RAM in the new server, so that'll get us out of the woods for a little while as MySQL will have a bit more space to breathe. With more RAM we should be able to use Memcache to take load off the DB for some repetitive tasks. With less requests to the DB it'll have less to do, and it'll be able to do it faster. There's also some indexes missing in the DB which should speed up the queries -- again, faster queries = faster page load = RAM is used for less time = more requests, faster.

All of this is a band-aid job. The real work comes when we write a new frontend.

Re: PALGN really slow? (1 year ago)

I can't - you're asking the wrong guy. I'm here to make 0's and 1's move around.

In order, no, and yes (it's a job, not my life).

There is a new server being commissioned (I believe its happening this week), so my job over the next couple of weeks will be to set it up to my satisfaction, then migrate code onto it and make sure it still runs. 0's and 1's.

Re: PALGN really slow? (1 year ago)

Whoops. Turns out when I get a notification from the forum that someone's replied to the thread, it doesn't send me another one unless I reply again.

A bug in the forum. Who knew?

So let's go back over the last few posts then...

I'm a PHP developer by trade. Over the last 10 years I've worked at Melbourne IT (funny story, I helped launch .biz in a way), Fairfax Media, Globe Shoes and I'm currently working at a particular national broadcaster in Sydney.

Not to put too fine a point on it, but I've worked on sites where thousands of users sign in each hour. I've seen systems under heavy load (and if I'm honest, caused some of it from time to time). This is kind of like Sleepy Hollow in comparison.

The CMS is the backend tool that allows the staff to manage the content of the site. How the content is stored and updated will change, and we'll build a new frontend focused on getting that content onto a page as fast as possible. The forum is also a part of the frontend, and we'll be looking at it along the way.

Thanks, your post reads like you just want to have a whinge. Maybe you forgot to add "No offence, but" to the front of that sentence. Or maybe you didn't. #justsayin

I'm not getting into he-said/she-said because it doesn't help anyone. We're on page 5 of this thread, I think it's pretty obvious to everyone that we know there are issues.

Re: PALGN really slow? (1 year ago)

Yes.

We started some discussions around the time that message was posted, but I've had a few other things get in the way (family holiday, changing jobs, you know - the little stuff!).

Re: PALGN really slow? (1 year ago)

It's too early to discuss anything specific about the proposed new system at this stage. It's an idea (wrapped in an enigma, enclosed in a plan, etc...).

Don't be concerned about changes and differences - my goal is to ensure we can provide the services and functions that users want, and to do it in such a way that the site doesn't need to sit down for a rest after serving a page

Re: PALGN really slow? (1 year ago)

Hi everyone. Here's some stuff you might appreciate.

Firstly, a brief introduction - I'm Jason. I've actually been a member of PALGN for a while (~2006), though it's fair to say I haven't really been an active user. I'll be taking on some technical duties behind the scenes.

To tackle the elephant in the room: yes, there are some issues with PALGN's software and its current hardware infrastructure. I've observed some of the the high load periods users experience at various times during the day, and collected some metrics on what's actually happening to the system at these times.

We'll be tackling these issues in two ways:

Firstly, we'll be migrating everything to a new server shortly. It'll have a beefier CPU and more RAM and all that good stuff, so things should start moving faster. This should happen in the short-term.

Secondly, after this server migration is complete, we'll be looking at moving PALGN onto a new CMS backend that will better position us for the future. This is a longer-term goal. While we build it, we'll band-aid the existing system if it needs it, but we'll be focused on our new system.

In summary, we hope to have some server performance gains soon, and we'll be addressing functionality issues as we build our new system.

I'll be around and I'll look in from time to time. You can find me on The Twitter if you need, I'm @JM77. Don't be strangers.

Thanks for listening, reading and (hopefully) generally not flaming me. Much. Yet.

Jason

Re: Virtual Console (6 years ago)

Not to mention the DS games that Rare's been making... just wanted to point that out since everyone seems to think Rare doesn't touch Nintendo hardware any more...

(obtopic: Bottom line, Diddy is Nintendo's IP)

Re: Wii imports? (6 years ago)

I'm keeping an eye on dvdboxoffice.com, bought a lot of Cube games from them in the past. Console price is good (well, what I expect anyway), and the game prices are good.

DVDBO's free shipping (on DVDs, etc) isn't usually very fast, but its free and the games work out cheap. You can pay extra for fast shipping though.

What concerns me though, is what services or wii channels I might be unable to access if I buy a US Wii instead of an Aussie one.

I'm also heading to Hong Kong over Christmas so would consider buying a console there, but I want to know what the regional implications would be, since I have a JPN/USA cube and a lot of US cube games (and a couple of japanese). Anyone got any clue on this??