The Murkworks

Power failure over, Murkworks North

Tue, 29 Apr 2008 15:42:55 GMT

Murkworks North lost power sometime around 5AM in the morning on Tuesday, 20 April 2008; power was restored and servers brought online around 8:30AM. Checks are ongoing but there is no early evidence of data loss; however, some email may be delayed.

Serious Flash player vulnerability; recommend all users patch immediately!

Thu, 17 Apr 2008 00:21:04 GMT

If you are unaware of this recently published Adobe Flash Player exploit, you need to update your version of the player in all browsers on all operating systems now. Here is Adobe's support bulletin, which includes instructions on how. This is not just a Windows exploit, or an Internet Explorer exploit; it exists in all versions of Flashplayer and allows execution of arbitrary code. So far, only demonstration code is known to exist, so you have a window of opportunity to get your systems patched before Russian mobsters are p0wning your box.

If you don't know whether you have Flash player installed, you almost certainly do. If you've ever been to Homestar Runner to read Strongbad Email, for example, and seen anything at all, you have Flash player.

mail server software upgrade completed; hardware upgrade problems

Sat, 02 Feb 2008 09:24:22 GMT

Murknet mail services are back online; pop3 restored; new services supported include pop3s, imap4, and imap4s. Please use SSL (encrypted) channels for mail downloads if possible. (Most email clients support this.) On the problematic side, the server is ignoring the RAM we added, and we don't know why. More investigation will follow.

Downtime tonight for mail services upgrade

Fri, 01 Feb 2008 17:09:59 GMT

Mail services will be offline tonight probably after 8pm PST for a scheduled server upgrade - unless this timing is a problem, in which case let us know right now. Other services should remain unaffected.

MurkMUSH back online

Thu, 31 Jan 2008 00:20:15 GMT

Anna got the MurkMUSH backup extracted; it's from 14 January, so if you've done anything since then, it's lost. Sorry - but at least we're back up, yay.

MurkMUSH offline

Wed, 30 Jan 2008 23:53:20 GMT

MurkMUSH is temporarily offline due to database corruption. We'll be pulling a backup of the database online later today, but can't get to it right now for time. ETA for recovery sometime this evening.

Web services restored.

Fri, 25 Jan 2008 02:03:03 GMT

murkworks.net web services restored. Finally.

Attempt #2 unsuccessful

Thu, 24 Jan 2008 06:56:52 GMT

Well, we had a very nice offer of a replacement server machine (dual PIII 600 - old, but more than we'd need), and we tried installing it tonight - but, sadly, it won't support more than one IDE/ATA drive, and if you put a master/slave pair on, it gets confused and won't boot. So tomorrow I buy a little bunch of new hardware :( and give it another go.

Update; murknet web server still offline

Wed, 23 Jan 2008 04:42:42 GMT

lodestone suffered significantly more damage than originally hoped; we're going to be building a new box off the (we're pretty sure) still-in-good-order hard drives tomorrow. sadly, this costs significantly more dollars, as we can't just pop in a new power supply (or even new power supply, motherboard, and possibly processor). we can't even use the same case - we're hoping for the drives and net card. and yes, we are quite unhappy about this, but, well, we need a web server. eta for restart unknown, hopefully tomorrow evening.

murkworks.net web services offline; power supply failure

Tue, 22 Jan 2008 18:34:27 GMT

murkworks.net web services are offline; we just had a power supply failure, or, more specifically, a power supply fan failure, and we overheated. I'll be off to get a replacement fan in a little bit. ETA on return probably 4h.

newmoon (mail.murkworks.net) online 19:00 PST (on schedule)

Sat, 22 Dec 2007 03:14:44 GMT

Mail services are back online, but while I did wire the new fan into the power supply, the software report about the RAM on the motherboard, like the cake, was a lie. So the RAM upgrade is on hold until we can get more RAM that actually fits into the @#*$&(!!! machine, as is the round of software installs we'd planned tonight. OH WELL!

newmoon (mail.murkworks.net) downtime tonight

Fri, 21 Dec 2007 21:01:26 GMT

All Murknet mail services will go offline at 6pm tonight - Friday, 21 December 2007 - for RAM upgrade and cooling fan replacement. Downtime should be approximately one hour, but we'll try to do better than that.

Upcoming downtime for murknet mail server TBD

Wed, 12 Dec 2007 01:53:43 GMT

We know what kind of RAM we need; now we just have to buy and install it. If you have a specific time you absolutely need mail services, please let us know in email; we'll be scheduling downtime for the RAM upgrade sometime within the next week.

Email services downtime, 9 December 2007, 20:00 PST

Sun, 09 Dec 2007 19:48:36 GMT

Unless someone has an important objection, the murkworks.net mail server will be taken offline at 8pm (20:00) PST tonight, Sunday, 9 December 2007, for configuration examination in advance of a memory upgrade. Downtime should be less than one hour. IF THIS IS A SEVERE PROBLEM FOR YOU, PLEASE RESPOND IMMEDIATELY. Thanks.

Widespread flooding, continued heavy rain, winds expected to pick up

Mon, 03 Dec 2007 23:07:51 GMT

It's quite the day here in Cascadia as our neighbour town of Bothell - Murkworks North is in Kenmore - has declared a flooding state of emergency and has started some mandatory evacuations; our other neighbour of Lake Forest Park has opened a relief centre in the public library; the Burke-Gilman trail is underwater in some locations; parts of I-405, I-5, SR-522, SR-527, and many other highways and roads are closed and/or underwater; many train routes are out, and busses are delayed; but so far, we still have power! Despite this, please be aware of possible power outages and the shutdowns that will follow immediately behind; wind gusts of up to 50mph were still in the prediction last I checked.

High winds today at Murkworks North

Mon, 12 Nov 2007 16:15:13 GMT

High winds and rain forecast today at Murkworks North, with the storm that's on the way already taking out power in parts of British Columbia. We'll only go offline if we actually lose power, of course, but given that last month's dinky windstorm knocked out the entire northlake corridor, I won't be surprised if it happens. (Power has flickered a few times already.) On the plus side, if we do lose power, it'll be a good time to replace that fan in lodestone's power supply.

More if and when we get it...

Power outage, then recovery; lodestone needs new fan

Fri, 19 Oct 2007 05:03:46 GMT

Power outage occurred around 17:40, power was restored around 19:30. No software problems noted at startup, but let us know if anything seems to be going pear-shaped on you. Lodestone's power supply fan seems to have picked up a significant bit of a buzz, expect downtime to replace that before it seizes up and takes the power supply with it.

bit of a storm moving in...

Thu, 18 Oct 2007 18:38:50 GMT

There's a bit of a windstorm moving in, carrying heavy rain with it; the rain shouldn't be a problem and the winds aren't supposed to be strong enough to cause concern either, but we've already had three brownouts, one of which was pretty substantial, so I'm not banking on that. If we go offline abruptly, well, that's why.

anticipated downtown, mail server (newmoon)

Mon, 13 Aug 2007 16:08:59 GMT

We are anticipating downtime for our primary email server (newmoon) later this week in order to open it and find out what kind of RAM expansion it takes, as I haven't found the motherboard manual. We'll put it back up, go buy ram, and then take it offline again for the upgrade. The two downtime segments should not be particularly long. The reason for the upgrade is to expand capacity to host journaling software.

Other than that, there are no known murknet issues at this time. Yay!

murknet primary router, DNS failure (update 6)

Wed, 20 Jun 2007 07:22:06 GMT

Shortly after the last update, spazzkat came up with a 100mbps hub, which we installed immediately, helping our throughput. Yesterday, I repaired our backup 100/10mbps switching hub, which has only barely enough ports for our server rack, but is still good to have as a backup. The regular switch has been sent off to Linksys for warranty service. Finally, packet-filtering resumed late Sunday night. I may write up an event post-mortum, but if I do, it will be on my personal journal, solarbird, instead of here, because this appears on our home/status page and I like to keep these entries relatively small.

If you are still experiencing service loss of murknet functionality, please let us know so that we can try to figure out what we missed.

murknet primary router, DNS failure (update 5)

Mon, 18 Jun 2007 04:48:07 GMT

We have partially restored service. Thanks to fenton in particular for the magic key, which is to say, knowing that the ip_forwarding default had changed and how to set it back. The routing issue we saw mentioned elsewhere on the net is real, but does not affect our configuration, so that's not going to be an issue. We will be slower than usual, as our server backbone hub joined the chorus of hardware LEAVING EARTH FOREVER FOR NO GOOD RAISIN, and we currently have an elderly 10mbps hub out of our antique hardware rack standing in. (Seriously. It supports coax. But it's working.)

More, as usual, when we have it.

murknet primary router, DNS failure (update 4)

Mon, 18 Jun 2007 02:51:40 GMT

Via a collection of tools and new hardware (and now our backbone switch has died - this is one hell of a weekend, I gotta tell you), we've gotten door to talk to the net. And door to talk to the LAN. However, it is refusing to deliver packets across to each other. I have found terrifying notes on t3h intarw3bz that this is horribly, horribly broken in debian 4.

Also, dselect and dpkg are doing my favourite trick again - the one where dselect sees packages available that dpkg doesn't, but won't let me use it to get the packages because it's decided there's a package descrepency that does not, in fact, exist, and wants to "fix" it by breaking our config. I don't know why dselect can see these packages and dpkg can't. But the more important thing right now is getting door to relay packets - somehow.

murknet primary router, DNS failure (update 3)

Sun, 17 Jun 2007 20:07:31 GMT

We have finally had something go right and have managed to wedge new data into door. This has not been trivial. I have to take this last machine off the net to go to the next step. More when we're back on the air.

murknet primary router, DNS failure (update 2)

Sun, 17 Jun 2007 08:06:46 GMT

Still down. We've given up on trying to get any of the 2.4.x net drivers working; everything we have cards for or could get cards for fails, often with unresolved externals (what the fuck! what the fucking fuck!) in the distribution core fileset. Something is seriously, seriously wrong with the state of system files on our router/gateway/etc machine and we are well and truly pooched as a result.

I'm pulling down a network install CD and Anna is pulling down a complete set of install CDs. If I can get a network install CD successfully pulled down tonight, and if I can keep door's CD-ROM working (not a given - it already wedged itself once today), we may be able to be online on Sunday sometime in the afternoon. However, if not, it could be a few days.

As always, more as we get it.

murknet primary router, DNS failure (update)

Sun, 17 Jun 2007 03:39:20 GMT

We have a serious continuing issue with networking affecting both DNS and routing. At this point we are still completely offline and still have no ETA, but are continuing to work on the problem. We are considering applying cannon fire to door in the event that door wishes to retire. Seriously, nothing is working right, and Debian experts we know are currently at status mystified, which is never good, since they work on the kernel. So. Updates as we have them.