RPTools.net

We had an outage of the site for about 24 hours this weekend. The cpu and memory use of the site spiked and stayed pegged for almost an hour and HostGator had to disable the site. Then it took me a little time to work with them regarding what the issues were and the best steps to correct them.

The primary culprit seems to be the gallery software. I have renamed the directories that contain the software and the data to avoid them from being used. During the week I'll take a look and see if: (1) the software is upgradeable, (2) whether the upgrades provide a performance improvement, (3) whether the work involved is justified to keep the gallery, and (4) whether there are forum users who link directly to the gallery for their avatar image or whether there are posts that link directly to the gallery.

Trevor has indicated in the past that he wanted to wipe it out -- there are plenty of file/image sharing sites on the web so does RPTools really need to provide one?

That will be part of the evaluation process this week. I may be able to bring the gallery back up for a short amount of time, but don't expect it. I might simply create a read-only directory of images (if I can) or simply leave it turned off (if I have to).

The good thing about all this is that the inode count will drop below the threshold where the site receives automatic backups from HostGator. The site hasn't been below that threshold for a few years now (!) which means no backups of the data other than the ones I make when I'm about to do a forum upgrade or perform similar maintenance.

I'll post here again when I have more information, but it won't be until 8/26 because my weekend is pretty full.

Thanks for the update, Az, and thanks for working the problem. I'll be sad to lose the gallery but it's far preferable to losing the site!

Ah crap I knew it was bad idea to have images linking back to this site, no offense. It just seems like a better idea to host them on actual image hosts. Ya literally when I first started using the fancy image buttons for macros I was like....hmmm what happens if the server goes down or becomes unreachable.

How big is the gallery ?

there are some pretty nice things on the gallery it would be a waste to can them. Maybe theres a possibility to zip them up and put them on some torrent, download site or such?

Yeah, it would be a shame to throw it all a way.

An archive should be possible to host somewhere.

I vote for a torrent that would be nice, don't have to front the cost.

wolph42 wrote:there are some pretty nice things on the gallery it would be a waste to can them. Maybe theres a possibility to zip them up and put them on some torrent, download site or such?

CoveredInFish wrote:Yeah, it would be a shame to throw it all a way.

An archive should be possible to host somewhere.

Volomon wrote:I vote for a torrent that would be nice, don't have to front the cost.

Yes! I support this idea too. Please. I like the torrent distribution.

PS: If it is possible i prefer a compression in multiple volumnes (1 GB or so).

DJuego

Please just give us enough time to get our stuff off there. I don't have back ups of much of my stuff and put tons of hours into that.

..not quite sure how the access works though as I'm not tech saavy.. .php what?



jh

I still haven't looked at this, life being busy and all. But yes, I understand it would be a bummer to lose it.

It looks like the next couple of days are going to be full of "non-billable-but-have-to-do" work, so Friday at the earliest before I can look into this.

Some initial investigations indicate there was some kind of injection attack (no research yet) which might explain the cpu spike. Maybe a gallery update/upgrade would be enough. We'll see.

I'll post again when I have something to report.

Thanks. I look forward to finding out what the verdict is. Moving all my sub-forum stuff is going to suck, but if it keeps the site going, it is worth it.

Thanks for following up!

jh

I just got another trouble ticket from HostGator that the cpu usage spiked up to 3380 seconds during the "past hour" (ie. 94% cpu).

I'm not sure I believe them though, because the load average was 0.10, 0.02, and 0.01. If you're a Unix person you know those are 1 minute, 5 minute, and 15 minute load averages. So how could there be 3380 seconds of cpu in an hour that has 3600 seconds, and yet the last 900 seconds has a load average of 0.01??

Unfortunately, my email addy is not the billing addy for the account so they send the ticket to Trevor who then has to forward it to me. We're going to change that. Today.

I forget what time it was, but there was some wonkiness going on this morning for a couple of minute. Pages were loading without the templates, but were fixed the next minute. Looking at my history, I think it happened just before 6:40 PST.

Another big outage. Same problem.

It seems that the gallery.rptools.net/main.php page was able to be accessed even though I put the rest of the gallery into maintenance mode. Apparently "maintenance mode" is so people can't add new stuff, but existing stuff can still be accessed.

I saw a ps(1) listing with about a half-dozen accesses to the gallery at once, each with 20-35% of the cpu (yes, I know that's more than 100%). I'm going to try upgrading the gallery to 3.0.9 (v3 is considered "post-beta" and we have v2.3 which is supposed to be the latest "stable" release). The current package was installed manually and I'm going to use the cPanel interface so it can keep it up to date for me. (I should do the same for the phpBB stuff, when I get a chance.)

I've got some work deadlines for Wednesday, so it'll likely be the end of the week before I can work on this unless I find myself with a few minutes. I did have an hour or so to tinker with MTL (the new MapToolLauncher) so who knows, maybe I'll get some more time...