|
|
You are viewing the most recent 25 entries.
6th November 2009
5:24pm: A second night of severe weather
Hi, it's the figs of the Murkworks again, and we're having a second straight night of not just unusual but downright freakish severe weather - half a centimetre of hail, FOOM FOOM FOOM rain, and lots and lots of lighting, which we just don't see here. The cats are terrified. Fortunately there's not much wind associated with it so far, which makes it a lot less likely we'll lose power, but it's a possibility. Be advised that you may need to log out quickly on short notice.
Current Mood:  excited
26th October 2009
8:32am: Winter storm advisory
We have our first winter storm advisory of the year, and with it, a series of brownouts that have hit what we internally call "Stage 2." (Stage 1: First alarms tripped. Stage 2: Multiple alarms, typically plus momentary darkness because we lose the lights. Stage 3: Extended power losses less than two minutes. After that, we shut down.) Hopefully it won't get any worse than this, but it could. Be prepared for power-related shutdowns on short notice.
Current Music: Birdhouse in Your Soul | They Might Be Giants
29th July 2009
2:30pm: Server room at 30.1C
As per the previous update, we're dealing with record-smashing heat today, and Murkworks North is sadly not air-conditioned. Through a variety of tricks, we've been keeping the temperature down from 39.1°C/102.3°F outside, but the server room has just passed 30°, a bit after 2pm. We will begin shutting down services at 34°C. Be prepared for shutdown without specific notice. I'll be tracking temperature (of course) and will notify again at 32°C.
Current Mood:  hot
27th July 2009
8:29pm: Heat-related shutdowns possible (if not likely)
We're experience a record-setting heat wave here around the Nightmare Factory, and while we're pretty good at heat management, we do not have full A/C, and the Murkworks may have to shut down some servers during peak heat times of the day. This will happen with relatively little notice. The order of shutdown will be, as necessary: backup services, mail services, web services, then DNS and complete network, in that order. We fully expect to take at least backup services offline Tuesday and Wednesday (28/29 July), with predicted highs around 37°C on Wednesday. It is likely that we will take down mail services as well on Wednesday, but we will work to avoid it. Temperatures are expected to return to ranges more approximating normal on Thursday and Friday, or so we hope. eta: We just broke 30° in the server room. Backup services are going offline now.
31st March 2009
11:37am: Unexpected winds
It's unexpectedly windy today and Murkworks North has experience a series of short power flickers substantial enough to trigger seven (and counting) Level 1 UPS alarms. (We have alarms that trigger at different degrees of interruption. Level 1 is highly transient and includes brownouts, Level 2 is substantial but less than one second, Level 3 is full power failure.) Accordingly, we may have to shut down servers with very little to no warning. Consider this your heads-up.
22nd February 2009
10:19pm: mail, web servers upgraded
lodestone and newmoon have both been upgraded with a minimum of fuss. so what the hell's the point of that? they've taken all the fun out of it with this whole shit-just-working thing. where's the joy of seething hate and unthinking, blinding rage? i miss it.
11:21am: Heads up: Imminent kernel and OS upgrades on murkworks.net
Hi folks, annathepiper here. Debian 5.0 has come out so we'll be working to upgrade the murkworks server core to the new release. This means we also have to upgrade the system kernel on both newmoon and lodestone. All the current backup files are being burned down to DVD so nobody should lose any data over this. But there will be reboots of servers today, with hopefully minimal downtime. I'll be emailing the users mailing list before these happen, and will also send out system notices. So if there's any particular time you don't want this to be happening, please drop a comment on this post or send me mail. More bulletins as events warrant.
Current Mood: busy
Current Music: Born Free | IBM 1403 Printer and Ron Mak
11:18am: Downtime this afternoon for kernel upgrades
Mail and web services will both being going offline briefly this afternoon for kernel upgrades as part of preparation for a second round of upgrades coming soon. Please be aware that both web and mail will be offline at time today with relatively little notice. Downtimes should, however, be short.
Current Mood: busy
Current Music: Born Free | IBM 1403 Printer and Ron Mak
13th January 2009
12:10pm: MurkMUSH back online
MurkMUSH is back online; we gave up on today's DB and rolled back to the (very recent) previous backup. It's safe to log on and hopefully nobody lost anything important.
10:39am: MurkMUSH offline
Yep, the MUSH is offline; a bad character got written to the DB and we're rebuilding the database now. It'll be back up once the database is rebuilt.
8th January 2009
5:52pm: Not affected by flooding
Here at Murkworks North, we're near the top of a giant hill (we own the top, actually, but I digress) and are accordingly not generally affected by flooding. Power has been stable, as well. We expect normal operations during the flooding.
6th January 2009
6:01pm: Wind and power flickers
Okay, so we've got high winds again at the Murkworks, and power has been flickering enough to trip UPS alarms a lot of the afternoon, so be warned that we might have to power down the server cluster with very little warning.
20th December 2008
11:41pm: Power hits, murkworks north
We're experiencing power drops here at Murkworks North; we'll stay online if we can, but if we lose power, we'll be offline pending restoration. Consider yourselves warned.
4th October 2008
4:09pm: Wind advisory, Murkworks North
There's an early-season windstorm here at Murkworks North today (4 October 2008), and power outages all throughout the Puget Sound region, so be prepared for possible sudden system shutdowns and other anomalous behaviour through the evening.
3rd July 2008
9:38am: Unstable weather, Murkworks North
Last night (2 July 2008) we started experiencing locally-severe thunderstorms in the central and northern Puget Sound region, along with the predictable power fluctuations which usually accompany this kind of weather. We're seeing more such activity today; there is currently a storm cell directly overhead. Accordingly, be prepared for possible power outages throughout 3 July.
29th April 2008
8:40am: Power failure over, Murkworks North
Murkworks North lost power sometime around 5 AM in the morning on Tuesday, 20 April 2008; power was restored and servers brought online around 8:30 AM. Checks are ongoing but there is no early evidence of data loss; however, some email may be delayed.
2nd February 2008
1:22am: mail server software upgrade completed; hardware upgrade problems
Murknet mail services are back online; pop3 restored; new services supported include pop3s, imap4, and imap4s. Please use SSL (encrypted) channels for mail downloads if possible. (Most email clients support this.) On the problematic side, the server is ignoring the RAM we added, and we don't know why. More investigation will follow.
1st February 2008
9:08am: Downtime tonight for mail services upgrade
Mail services will be offline tonight probably after 8pm PST for a scheduled server upgrade - unless this timing is a problem, in which case let us know right now. Other services should remain unaffected.
30th January 2008
4:19pm: MurkMUSH back online
Anna got the MurkMUSH backup extracted; it's from 14 January, so if you've done anything since then, it's lost. Sorry - but at least we're back up, yay.
Current Mood:  sleepy
Powered by LiveJournal.com
|