Site Outage from 8/23-8/24

New build announcements plus site news and changes.

Moderators: dorpond, trevor, Azhrei, Craig

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Site Outage from 8/23-8/24

Post by Azhrei »

We had an outage of the site for about 24 hours this weekend. The cpu and memory use of the site spiked and stayed pegged for almost an hour and HostGator had to disable the site. Then it took me a little time to work with them regarding what the issues were and the best steps to correct them.

The primary culprit seems to be the gallery software. I have renamed the directories that contain the software and the data to avoid them from being used. During the week I'll take a look and see if: (1) the software is upgradeable, (2) whether the upgrades provide a performance improvement, (3) whether the work involved is justified to keep the gallery, and (4) whether there are forum users who link directly to the gallery for their avatar image or whether there are posts that link directly to the gallery.

Trevor has indicated in the past that he wanted to wipe it out -- there are plenty of file/image sharing sites on the web so does RPTools really need to provide one?

That will be part of the evaluation process this week. I may be able to bring the gallery back up for a short amount of time, but don't expect it. I might simply create a read-only directory of images (if I can) or simply leave it turned off (if I have to).

The good thing about all this is that the inode count will drop below the threshold where the site receives automatic backups from HostGator. The site hasn't been below that threshold for a few years now (!) which means no backups of the data other than the ones I make when I'm about to do a forum upgrade or perform similar maintenance.

I'll post here again when I have more information, but it won't be until 8/26 because my weekend is pretty full.

User avatar
RPTroll
TheBard
Posts: 3159
Joined: Tue Mar 21, 2006 7:26 pm
Location: Austin, Tx
Contact:

Re: Site Outage from 8/23-8/24

Post by RPTroll »

Thanks for the update, Az, and thanks for working the problem. I'll be sad to lose the gallery but it's far preferable to losing the site!
ImageImage ImageImageImageImage
Support RPTools by shopping
Image
Image

Volomon
Cave Troll
Posts: 56
Joined: Fri Jul 02, 2010 4:43 am
Contact:

Re: Site Outage from 8/23-8/24

Post by Volomon »

Ah crap I knew it was bad idea to have images linking back to this site, no offense. It just seems like a better idea to host them on actual image hosts. Ya literally when I first started using the fancy image buttons for macros I was like....hmmm what happens if the server goes down or becomes unreachable.

User avatar
Dervish
Giant
Posts: 121
Joined: Thu Sep 15, 2011 7:52 pm

Re: Site Outage from 8/23-8/24

Post by Dervish »

How big is the gallery ?

User avatar
wolph42
Winter Wolph
Posts: 9999
Joined: Fri Mar 20, 2009 5:40 am
Location: Netherlands
Contact:

Re: Site Outage from 8/23-8/24

Post by wolph42 »

there are some pretty nice things on the gallery it would be a waste to can them. Maybe theres a possibility to zip them up and put them on some torrent, download site or such?

User avatar
CoveredInFish
Demigod
Posts: 3104
Joined: Mon Jun 29, 2009 10:37 am
Location: Germany
Contact:

Re: Site Outage from 8/23-8/24

Post by CoveredInFish »

Yeah, it would be a shame to throw it all a way.

An archive should be possible to host somewhere.

Volomon
Cave Troll
Posts: 56
Joined: Fri Jul 02, 2010 4:43 am
Contact:

Re: Site Outage from 8/23-8/24

Post by Volomon »

I vote for a torrent that would be nice, don't have to front the cost.

User avatar
DJuego
Cave Troll
Posts: 84
Joined: Fri Jun 01, 2007 1:54 pm

Re: Site Outage from 8/23-8/24

Post by DJuego »

wolph42 wrote:there are some pretty nice things on the gallery it would be a waste to can them. Maybe theres a possibility to zip them up and put them on some torrent, download site or such?
CoveredInFish wrote:Yeah, it would be a shame to throw it all a way.

An archive should be possible to host somewhere.
Volomon wrote:I vote for a torrent that would be nice, don't have to front the cost.

Yes! I support this idea too. Please. I like the torrent distribution.

PS: If it is possible i prefer a compression in multiple volumnes (1 GB or so).

DJuego

User avatar
emirikol
Dragon
Posts: 708
Joined: Sun Jan 13, 2008 5:52 pm
Location: Lakewood, CO North America
Contact:

Re: Site Outage from 8/23-8/24

Post by emirikol »

Please just give us enough time to get our stuff off there. I don't have back ups of much of my stuff and put tons of hours into that.

..not quite sure how the access works though as I'm not tech saavy.. .php what?

:)

jh
Yes, I'm a chiropractor. Gamer fitness at Hafner Chiropractic in Lakewood, CO: http://www.HafnerChiropractic.com

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Site Outage from 8/23-8/24

Post by Azhrei »

I still haven't looked at this, life being busy and all. But yes, I understand it would be a bummer to lose it.

It looks like the next couple of days are going to be full of "non-billable-but-have-to-do" work, so Friday at the earliest before I can look into this.

Some initial investigations indicate there was some kind of injection attack (no research yet) which might explain the cpu spike. Maybe a gallery update/upgrade would be enough. We'll see.

I'll post again when I have something to report.

User avatar
masterclif
Giant
Posts: 115
Joined: Fri Jul 16, 2010 4:22 pm
Contact:

Re: Site Outage from 8/23-8/24

Post by masterclif »

Thanks. I look forward to finding out what the verdict is. Moving all my sub-forum stuff is going to suck, but if it keeps the site going, it is worth it.
Image

User avatar
emirikol
Dragon
Posts: 708
Joined: Sun Jan 13, 2008 5:52 pm
Location: Lakewood, CO North America
Contact:

Re: Site Outage from 8/23-8/24

Post by emirikol »

Thanks for following up!

jh
Yes, I'm a chiropractor. Gamer fitness at Hafner Chiropractic in Lakewood, CO: http://www.HafnerChiropractic.com

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Site Outage from 8/23-8/24

Post by Azhrei »

I just got another trouble ticket from HostGator that the cpu usage spiked up to 3380 seconds during the "past hour" (ie. 94% cpu).

I'm not sure I believe them though, because the load average was 0.10, 0.02, and 0.01. If you're a Unix person you know those are 1 minute, 5 minute, and 15 minute load averages. So how could there be 3380 seconds of cpu in an hour that has 3600 seconds, and yet the last 900 seconds has a load average of 0.01??

Unfortunately, my email addy is not the billing addy for the account so they send the ticket to Trevor who then has to forward it to me. We're going to change that. Today. :)

User avatar
aliasmask
RPTools Team
Posts: 9024
Joined: Tue Nov 10, 2009 6:11 pm
Location: Bay Area

Re: Site Outage from 8/23-8/24

Post by aliasmask »

I forget what time it was, but there was some wonkiness going on this morning for a couple of minute. Pages were loading without the templates, but were fixed the next minute. Looking at my history, I think it happened just before 6:40 PST.

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Site Outage from 8/23-8/24

Post by Azhrei »

Another big outage. Same problem.

It seems that the gallery.rptools.net/main.php page was able to be accessed even though I put the rest of the gallery into maintenance mode. Apparently "maintenance mode" is so people can't add new stuff, but existing stuff can still be accessed.

I saw a ps(1) listing with about a half-dozen accesses to the gallery at once, each with 20-35% of the cpu (yes, I know that's more than 100%). I'm going to try upgrading the gallery to 3.0.9 (v3 is considered "post-beta" and we have v2.3 which is supposed to be the latest "stable" release). The current package was installed manually and I'm going to use the cPanel interface so it can keep it up to date for me. (I should do the same for the phpBB stuff, when I get a chance.)

I've got some work deadlines for Wednesday, so it'll likely be the end of the week before I can work on this unless I find myself with a few minutes. I did have an hour or so to tinker with MTL (the new MapToolLauncher) so who knows, maybe I'll get some more time...

Post Reply

Return to “Announcements”