Log in

No account? Create an account

The Murkworks

Recent Entries

You are viewing 25 entries, 25 into the past.

22nd February 2009

11:21am: Heads up: Imminent kernel and OS upgrades on murkworks.net
Hi folks, annathepiper here. Debian 5.0 has come out so we'll be working to upgrade the murkworks server core to the new release. This means we also have to upgrade the system kernel on both newmoon and lodestone.

All the current backup files are being burned down to DVD so nobody should lose any data over this. But there will be reboots of servers today, with hopefully minimal downtime. I'll be emailing the users mailing list before these happen, and will also send out system notices. So if there's any particular time you don't want this to be happening, please drop a comment on this post or send me mail.

More bulletins as events warrant.
Current Mood: busy
11:18am: Downtime this afternoon for kernel upgrades
Mail and web services will both being going offline briefly this afternoon for kernel upgrades as part of preparation for a second round of upgrades coming soon. Please be aware that both web and mail will be offline at time today with relatively little notice. Downtimes should, however, be short.
Current Mood: busy

13th January 2009

12:10pm: MurkMUSH back online
MurkMUSH is back online; we gave up on today's DB and rolled back to the (very recent) previous backup. It's safe to log on and hopefully nobody lost anything important.
10:39am: MurkMUSH offline
Yep, the MUSH is offline; a bad character got written to the DB and we're rebuilding the database now. It'll be back up once the database is rebuilt.

8th January 2009

5:52pm: Not affected by flooding
Here at Murkworks North, we're near the top of a giant hill (we own the top, actually, but I digress) and are accordingly not generally affected by flooding. Power has been stable, as well. We expect normal operations during the flooding.

6th January 2009

6:01pm: Wind and power flickers
Okay, so we've got high winds again at the Murkworks, and power has been flickering enough to trip UPS alarms a lot of the afternoon, so be warned that we might have to power down the server cluster with very little warning.

5th January 2009

1:01pm: Probability factor of 1:1; we have normality, I repeat, we have normality
Well, we're back up and running. We actually came back up - sorta - at 2am, in a controlled fashion, but didn't come back up entirely right in the router (or in the mail server) and so couldn't talk to the net, or, even if we could, get mail. I fixed mail before going out for an allergist appointment; annathepiper worked with technoshaman and got routing fixed.

4th January 2009

10:51pm: Complete Power Failure, Murkworks North
Murkworks North has has a complete power failure. All servers are being shut down NOW.

20th December 2008

11:41pm: Power hits, murkworks north
We're experiencing power drops here at Murkworks North; we'll stay online if we can, but if we lose power, we'll be offline pending restoration. Consider yourselves warned.

18th December 2008

4:54pm: Severe windstorm alert for Saturday, 20 December
There's a Severe Wind Watch in effect for the Puget Sound area this Saturday and Sunday, with forecast sustained winds as high as 75mph, and gusts up to 90mph. We're on the edge of the area and should be shielded from the worst of it, but if this watch turns out to be real - they're saying to stock up with food(!) - we will expect a multi-day power outage and will be just glad we aren't killed by falling trees.

4th October 2008

4:09pm: Wind advisory, Murkworks North
There's an early-season windstorm here at Murkworks North today (4 October 2008), and power outages all throughout the Puget Sound region, so be prepared for possible sudden system shutdowns and other anomalous behaviour through the evening.

3rd July 2008

9:38am: Unstable weather, Murkworks North
Last night (2 July 2008) we started experiencing locally-severe thunderstorms in the central and northern Puget Sound region, along with the predictable power fluctuations which usually accompany this kind of weather. We're seeing more such activity today; there is currently a storm cell directly overhead. Accordingly, be prepared for possible power outages throughout 3 July.

29th April 2008

8:40am: Power failure over, Murkworks North
Murkworks North lost power sometime around 5AM in the morning on Tuesday, 20 April 2008; power was restored and servers brought online around 8:30AM. Checks are ongoing but there is no early evidence of data loss; however, some email may be delayed.

16th April 2008

5:20pm: Serious Flash player vulnerability; recommend all users patch immediately!
If you are unaware of this recently published Adobe Flash Player exploit, you need to update your version of the player in all browsers on all operating systems now. Here is Adobe's support bulletin, which includes instructions on how. This is not just a Windows exploit, or an Internet Explorer exploit; it exists in all versions of Flashplayer and allows execution of arbitrary code. So far, only demonstration code is known to exist, so you have a window of opportunity to get your systems patched before Russian mobsters are p0wning your box.

If you don't know whether you have Flash player installed, you almost certainly do. If you've ever been to Homestar Runner to read Strongbad Email, for example, and seen anything at all, you have Flash player.

2nd February 2008

1:22am: mail server software upgrade completed; hardware upgrade problems
Murknet mail services are back online; pop3 restored; new services supported include pop3s, imap4, and imap4s. Please use SSL (encrypted) channels for mail downloads if possible. (Most email clients support this.) On the problematic side, the server is ignoring the RAM we added, and we don't know why. More investigation will follow.

1st February 2008

9:08am: Downtime tonight for mail services upgrade
Mail services will be offline tonight probably after 8pm PST for a scheduled server upgrade - unless this timing is a problem, in which case let us know right now. Other services should remain unaffected.

30th January 2008

4:19pm: MurkMUSH back online
Anna got the MurkMUSH backup extracted; it's from 14 January, so if you've done anything since then, it's lost. Sorry - but at least we're back up, yay.
Current Mood: sleepy
3:51pm: MurkMUSH offline
MurkMUSH is temporarily offline due to database corruption. We'll be pulling a backup of the database online later today, but can't get to it right now for time. ETA for recovery sometime this evening.

24th January 2008

6:02pm: Web services restored.
murkworks.net web services restored. Finally.
Current Mood: relieved

23rd January 2008

10:54pm: Attempt #2 unsuccessful
Well, we had a very nice offer of a replacement server machine (dual PIII 600 - old, but more than we'd need), and we tried installing it tonight - but, sadly, it won't support more than one IDE/ATA drive, and if you put a master/slave pair on, it gets confused and won't boot. So tomorrow I buy a little bunch of new hardware :( and give it another go.
Current Mood: frustrated

22nd January 2008

8:40pm: Update; murknet web server still offline
lodestone suffered significantly more damage than originally hoped; we're going to be building a new box off the (we're pretty sure) still-in-good-order hard drives tomorrow. sadly, this costs significantly more dollars, as we can't just pop in a new power supply (or even new power supply, motherboard, and possibly processor). we can't even use the same case - we're hoping for the drives and net card. and yes, we are quite unhappy about this, but, well, we need a web server. eta for restart unknown, hopefully tomorrow evening.
Current Mood: cranky
10:32am: murkworks.net web services offline; power supply failure
murkworks.net web services are offline; we just had a power supply failure, or, more specifically, a power supply fan failure, and we overheated. I'll be off to get a replacement fan in a little bit. ETA on return probably 4h.

21st December 2007

7:12pm: newmoon (mail.murkworks.net) online 19:00 PST (on schedule)
Mail services are back online, but while I did wire the new fan into the power supply, the software report about the RAM on the motherboard, like the cake, was a lie. So the RAM upgrade is on hold until we can get more RAM that actually fits into the @#*$&(!!! machine, as is the round of software installs we'd planned tonight. OH WELL!
Current Mood: pissed off
12:55pm: newmoon (mail.murkworks.net) downtime tonight
All Murknet mail services will go offline at 6pm tonight - Friday, 21 December 2007 - for RAM upgrade and cooling fan replacement. Downtime should be approximately one hour, but we'll try to do better than that.
Current Mood: geeky

11th December 2007

5:50pm: Upcoming downtime for murknet mail server TBD
We know what kind of RAM we need; now we just have to buy and install it. If you have a specific time you absolutely need mail services, please let us know in email; we'll be scheduling downtime for the RAM upgrade sometime within the next week.
Current Mood: busy
Powered by LiveJournal.com