fig

Server room at 30.1C

As per the previous update, we're dealing with record-smashing heat today, and Murkworks North is sadly not air-conditioned. Through a variety of tricks, we've been keeping the temperature down from 39.1°C/102.3°F outside, but the server room has just passed 30°, a bit after 2pm. We will begin shutting down services at 34°C. Be prepared for shutdown without specific notice. I'll be tracking temperature (of course) and will notify again at 32°C.
  • Current Mood
    hot hot
fig

Heat-related shutdowns possible (if not likely)

We're experience a record-setting heat wave here around the Nightmare Factory, and while we're pretty good at heat management, we do not have full A/C, and the Murkworks may have to shut down some servers during peak heat times of the day. This will happen with relatively little notice. The order of shutdown will be, as necessary: backup services, mail services, web services, then DNS and complete network, in that order. We fully expect to take at least backup services offline Tuesday and Wednesday (28/29 July), with predicted highs around 37°C on Wednesday. It is likely that we will take down mail services as well on Wednesday, but we will work to avoid it.

Temperatures are expected to return to ranges more approximating normal on Thursday and Friday, or so we hope.

eta: We just broke 30° in the server room. Backup services are going offline now.
fig

Unexpected winds

It's unexpectedly windy today and Murkworks North has experience a series of short power flickers substantial enough to trigger seven (and counting) Level 1 UPS alarms. (We have alarms that trigger at different degrees of interruption. Level 1 is highly transient and includes brownouts, Level 2 is substantial but less than one second, Level 3 is full power failure.) Accordingly, we may have to shut down servers with very little to no warning. Consider this your heads-up.
fig

mail, web servers upgraded

lodestone and newmoon have both been upgraded with a minimum of fuss. so what the hell's the point of that? they've taken all the fun out of it with this whole shit-just-working thing. where's the joy of seething hate and unthinking, blinding rage? i miss it.
fig

Heads up: Imminent kernel and OS upgrades on murkworks.net

Hi folks, annathepiper here. Debian 5.0 has come out so we'll be working to upgrade the murkworks server core to the new release. This means we also have to upgrade the system kernel on both newmoon and lodestone.

All the current backup files are being burned down to DVD so nobody should lose any data over this. But there will be reboots of servers today, with hopefully minimal downtime. I'll be emailing the users mailing list before these happen, and will also send out system notices. So if there's any particular time you don't want this to be happening, please drop a comment on this post or send me mail.

More bulletins as events warrant.
  • Current Music
    Born Free | IBM 1403 Printer and Ron Mak
fig

Downtime this afternoon for kernel upgrades

Mail and web services will both being going offline briefly this afternoon for kernel upgrades as part of preparation for a second round of upgrades coming soon. Please be aware that both web and mail will be offline at time today with relatively little notice. Downtimes should, however, be short.
  • Current Music
    Born Free | IBM 1403 Printer and Ron Mak
fig

MurkMUSH back online

MurkMUSH is back online; we gave up on today's DB and rolled back to the (very recent) previous backup. It's safe to log on and hopefully nobody lost anything important.
fig

MurkMUSH offline

Yep, the MUSH is offline; a bad character got written to the DB and we're rebuilding the database now. It'll be back up once the database is rebuilt.
fig

Not affected by flooding

Here at Murkworks North, we're near the top of a giant hill (we own the top, actually, but I digress) and are accordingly not generally affected by flooding. Power has been stable, as well. We expect normal operations during the flooding.