fig

Several Stage 0 and Stage 1 Power Alerts

It's a windy day up here at Murkworks North, and we're experiencing ongoing power alerts, and now have a wind advisory. (No kidding, guys. Thanks!) It is not uncommon for us to lose power under these circumstances; please be prepared to have systems shutdown with very little or no notice.

eta: Honestly, I've never seen power behaviour like this before. No full Stage 2 shit but power is just continually Stage 0ing on us over and over again. I'm amazed we're still up.
fig

Power behavior returns to normal

After a rather exciting day of power adventures, things seem to have returned to normal here at Murkworks North, and hopefully operations will continue as typical for now. As always, watch this account for bulletins should need arise.
fig

Level 2 power alert

We're seeing repeated deep brownouts here at Murkworks North this morning (2011/11/14). Please be aware that we could lose power with very little warning under these conditions, and be ready to log off or otherwise disconnect at any time. Further updates as the situation warrants...
fig

Last call: All 209.20.199 addresses going away

Everything related to 209.20.199.0 goes away tomorrow, June 30th. That includes gateway.murkworks.net (209.20.178.33), which technically wasn't ours but was managed by us. If you're still using any IP in that rage, you're gonna stop whether you like it or not by midnight tomorrow, and possibly a little before that.

Fairwell, ancien régime. I'd hoped we'd make it to IPv6, but as I've been telling myself: better online with a /30+1, than offline with a Class C, no matter how spiffy that Class C might've been.
fig

As 209.20.199.0 winds down...

I've just pulled a bunch of wires out of 209.20.199.0 support. If you haven't moved to the new IP addresses, things will seriously stop working now. (We're still answering DNS queries on it, but not a lot else.) There's very little traffic left on the DSL uplink, but not zero, and it should be zero!

Experimentally, I have attempted to re-register ns.murkworks.net as a nameserver in the new address space. Hopefully this might even work!

Last call, everyone. MURKWORKS--SBLK1, it's been nice knowing you.
#
# The following results may also be obtained via:
# http://whois.arin.net/rest/nets;q=209.20.199.0?showDetails=true&showARIN=false
#

Murkworks MURKWORKS--SBLK1 (NET-209-20-199-0-1) 209.20.199.0 - 209.20.199.255
Infinity Internet, Inc. NET-209-20-128-0-1 (NET-209-20-128-0-1) 209.20.128.0 - 209.20.255.255
Snif.
  • Current Music
    Sad Muppet | CRIME and the Forces of Evil
  • Tags
fig

Move updates; mail, particularly.

Some name servers out there are being particularly lame about updating their murkworks.net information, and are using the death date instead of time-to-live before updating. We're equipped for that on the web side - the server is answering to both the new and old address sets for most people - but the mail server, it turns out, has the wrong flavour kernel and balks when we try, so it's only answering to the new address.

If you are seeing this behaviour, please flush your DNS cache. Or if you're still accessing the webserver via the old IP, do that too. You should see much better performance - both in speed and reliability - as soon as you're seeing the new IP addresses.

If you do this and still can't talk to the mail server - like, say, all AT&T customers as far as we can tell - please switch to "mailx.murkworks.net" which is a new name introduced and therefore must be freshly looked up, because it can't be in your cache. That'll connect you to newmoon (the mail server) over the new pipes.

We'd've taken more time and made newmoon support both IPs except we were losing 40% of packets over the DSL again yesterday - burn in hell, Pioneer - and we just said fukkit, let's go. Really, this isn't going too badly; I wasn't even sure we'd be able to support both IP sets on the LAN like we are, because of questions about routing on the Comcast side, but turns out their little router box is smarter than they thought, and it works great! If newmoon had the right kernel, this would've been utterly transparent. As it is, this isn't too bad, and is moving reasonably quickly.

More as we have it...
  • Current Mood
    accomplished
  • Tags
fig

Major changes coming to murkworks.net

As you may be aware, the ongoing problems with our upstream link to Infinity Internet have continued to mostly remain unresolved; we go through extended bursts of time with 70% packet loss, falling offline, particularly when it's sunny, because Verizon and now Frontier will not maintain their infrastructure. IINet is not supporting other transports at this time, so we're stuck with Frontier and DSL if we want to keep talking with them.

Given that we're trying to run an assortment of professional websites here, and given that our throughput rates are no longer up to modern standards and cannot be either relied upon or upgraded in place, we are forced to move to a new upstream provider.

This means prying our Class C out of my cold, dead hands, which makes me sad. We'll be squeezing our server farm into five fixed IP addresses and all the workstations onto DHCP and address translation, and honestly I'm not thrilled about that, but, well, one does what one must. Offline with a Class C is not better than online with a /30+1.

Over the next as-quickly-as-I-can-manage-it days, we're going to be moving servers from the old net to the new net. If things go spectacularly well, we'll be able to do this quickly and quietly with a minimum of downtime, and support both address spaces during the transition through multiple interfaces and routing tricks. On the other hand, if things go as expected, it'll be rocky for a couple of weeks. If you're addressing anything on our network by IP address, stop now, and start using hostnames. The 209.20.199 hostblock will be history within a month, so it's time to join the... um... 1980s... and start using hostnames. Seriously, like right now.

If you have files on our servers, either newmoon or lodestone, now would be a very good time to doublecheck that you have backups. Make sure in particular that you have off-site copies of anything critical. You may not be able to get to them at times during the transition, and there may and may not be notice of offline time if things go particularly badly.

If you use us as a DNS server, you should already have received mail from us! If you have not, contact me IMMEDIATELY.

If you have virtual hosting with us, mail, webhosting of any sort, and so on, we hope that this will be transparent and you won't have to do anything at all, as long as you aren't hardcoding IP addresses in anything. If you are, stop now, and start using names. For theoretical best transportability, use mail.murkworks.net and web.murkworks.net, but really all names get treated about the same.

Thanks, and hopefully we'll get through this with as little pain as possible, with a glorious new era of something vaguely like modern net speeds and stability at the end of the process. Wish us luck.
  • Current Mood
    hopeful hopeful
fig

Newmoon back up

Okay folks, newmoon is talking to the world again. Mail seems to be coming in and going out. Mailing lists are apparently working again, and it is now once more possible to ssh into the box.
It took solarbird getting home to help me beat the box into shape, though. And this is what we had to do:


  • Dara discovered the cd drive in the box wasn’t actually properly plugged in. Plugged that back in properly, only to discover that it was old and crotchety enough that it was refusing to read the rescue disc I’d created

  • Tried to burn a second rescue disc, but the old and crotchety drive wouldn’t read that either, so Dara swapped the old drive out for a different one we had

  • Using our backups from earlier in the week, I grabbed a screwed up copy of a file and got us back to the point of being to log in

  • Once we were able to log in, I had to reinstall a package called libssl, which I had to do by manually grabbing the .deb file since Debian’s actually had an OS update and apt-get wouldn’t let me get the version I wanted (for those of you for whom that actually makes sense)


Long story short, those of you who rely on newmoon, please, PLEASE check your stuff and make sure that everything looks okay. If you have files locally stored on the box please ssh in and check them. If you’re getting mail from newmoon, please doublecheck that your mail clients are behaving as you expect.

Please report any issues to Anna or Dara. Drop a comment on this post, or else send us email at either our murkworks or gmail addresses.
fig

Newmoon damaged by power hit; murkworks mail services down

We just had a couple of power spikes at the Murk, like we do. This time though they appear to have taken out one of our UPSes--and it was the one that newmoon, our mail server, was plugged into. When I tried to bring newmoon back up, it worked for a while and then hung. I am not currently able to log into the system to restart its services.

So this means that as of this afternoon, mail services on murkworks.net are down. This impacts not only personal accounts but the various mailing lists we run. Please spread the word if you're on any of our mailing lists, and if you have backup personal email accounts, let people know they should use them this weekend if they need to get in touch with you.

solarbird or I will post here when we have more updates. Watch here, either of our journals, or my Twitter or Facebook accounts.

Sorry about this, folks. :/

- Anna
fig

Intermittent brownouts at Murkworks North

We're seeing intermittent brownouts this morning at Murkworks North; at the moment, they're just enough to cause lights flicker and WiFi interruptions locally. All the servers are, of course, on wired connections. If it stays at only this level, we'll have no problems staying online, but watch for the possibility of short-notice shutdowns in the event of power outage.
  • Current Music
    Here to Go | DEVO