?

Log in

No account? Create an account

The Murkworks

Recent Entries

You are viewing the most recent 25 entries.

25th December 2011

12:12pm: Several Stage 0 and Stage 1 Power Alerts
It's a windy day up here at Murkworks North, and we're experiencing ongoing power alerts, and now have a wind advisory. (No kidding, guys. Thanks!) It is not uncommon for us to lose power under these circumstances; please be prepared to have systems shutdown with very little or no notice.

eta: Honestly, I've never seen power behaviour like this before. No full Stage 2 shit but power is just continually Stage 0ing on us over and over again. I'm amazed we're still up.

14th November 2011

7:52pm: Power behavior returns to normal
After a rather exciting day of power adventures, things seem to have returned to normal here at Murkworks North, and hopefully operations will continue as typical for now. As always, watch this account for bulletins should need arise.
11:00am: Level 2 power alert
We're seeing repeated deep brownouts here at Murkworks North this morning (2011/11/14). Please be aware that we could lose power with very little warning under these conditions, and be ready to log off or otherwise disconnect at any time. Further updates as the situation warrants...

29th June 2011

1:55pm: Last call: All 209.20.199 addresses going away
Everything related to 209.20.199.0 goes away tomorrow, June 30th. That includes gateway.murkworks.net (209.20.178.33), which technically wasn't ours but was managed by us. If you're still using any IP in that rage, you're gonna stop whether you like it or not by midnight tomorrow, and possibly a little before that.

Fairwell, ancien régime. I'd hoped we'd make it to IPv6, but as I've been telling myself: better online with a /30+1, than offline with a Class C, no matter how spiffy that Class C might've been.

24th June 2011

11:55pm: As 209.20.199.0 winds down...
I've just pulled a bunch of wires out of 209.20.199.0 support. If you haven't moved to the new IP addresses, things will seriously stop working now. (We're still answering DNS queries on it, but not a lot else.) There's very little traffic left on the DSL uplink, but not zero, and it should be zero!

Experimentally, I have attempted to re-register ns.murkworks.net as a nameserver in the new address space. Hopefully this might even work!

Last call, everyone. MURKWORKS--SBLK1, it's been nice knowing you.
#
# The following results may also be obtained via:
# http://whois.arin.net/rest/nets;q=209.20.199.0?showDetails=true&showARIN=false
#

Murkworks MURKWORKS--SBLK1 (NET-209-20-199-0-1) 209.20.199.0 - 209.20.199.255
Infinity Internet, Inc. NET-209-20-128-0-1 (NET-209-20-128-0-1) 209.20.128.0 - 209.20.255.255
Snif.
Current Mood: sad

17th June 2011

8:45am: Move updates; mail, particularly.
Some name servers out there are being particularly lame about updating their murkworks.net information, and are using the death date instead of time-to-live before updating. We're equipped for that on the web side - the server is answering to both the new and old address sets for most people - but the mail server, it turns out, has the wrong flavour kernel and balks when we try, so it's only answering to the new address.

If you are seeing this behaviour, please flush your DNS cache. Or if you're still accessing the webserver via the old IP, do that too. You should see much better performance - both in speed and reliability - as soon as you're seeing the new IP addresses.

If you do this and still can't talk to the mail server - like, say, all AT&T customers as far as we can tell - please switch to "mailx.murkworks.net" which is a new name introduced and therefore must be freshly looked up, because it can't be in your cache. That'll connect you to newmoon (the mail server) over the new pipes.

We'd've taken more time and made newmoon support both IPs except we were losing 40% of packets over the DSL again yesterday - burn in hell, Pioneer - and we just said fukkit, let's go. Really, this isn't going too badly; I wasn't even sure we'd be able to support both IP sets on the LAN like we are, because of questions about routing on the Comcast side, but turns out their little router box is smarter than they thought, and it works great! If newmoon had the right kernel, this would've been utterly transparent. As it is, this isn't too bad, and is moving reasonably quickly.

More as we have it...
Current Mood: accomplished

14th June 2011

2:43pm: Major changes coming to murkworks.net
As you may be aware, the ongoing problems with our upstream link to Infinity Internet have continued to mostly remain unresolved; we go through extended bursts of time with 70% packet loss, falling offline, particularly when it's sunny, because Verizon and now Frontier will not maintain their infrastructure. IINet is not supporting other transports at this time, so we're stuck with Frontier and DSL if we want to keep talking with them.

Given that we're trying to run an assortment of professional websites here, and given that our throughput rates are no longer up to modern standards and cannot be either relied upon or upgraded in place, we are forced to move to a new upstream provider.

This means prying our Class C out of my cold, dead hands, which makes me sad. We'll be squeezing our server farm into five fixed IP addresses and all the workstations onto DHCP and address translation, and honestly I'm not thrilled about that, but, well, one does what one must. Offline with a Class C is not better than online with a /30+1.

Over the next as-quickly-as-I-can-manage-it days, we're going to be moving servers from the old net to the new net. If things go spectacularly well, we'll be able to do this quickly and quietly with a minimum of downtime, and support both address spaces during the transition through multiple interfaces and routing tricks. On the other hand, if things go as expected, it'll be rocky for a couple of weeks. If you're addressing anything on our network by IP address, stop now, and start using hostnames. The 209.20.199 hostblock will be history within a month, so it's time to join the... um... 1980s... and start using hostnames. Seriously, like right now.

If you have files on our servers, either newmoon or lodestone, now would be a very good time to doublecheck that you have backups. Make sure in particular that you have off-site copies of anything critical. You may not be able to get to them at times during the transition, and there may and may not be notice of offline time if things go particularly badly.

If you use us as a DNS server, you should already have received mail from us! If you have not, contact me IMMEDIATELY.

If you have virtual hosting with us, mail, webhosting of any sort, and so on, we hope that this will be transparent and you won't have to do anything at all, as long as you aren't hardcoding IP addresses in anything. If you are, stop now, and start using names. For theoretical best transportability, use mail.murkworks.net and web.murkworks.net, but really all names get treated about the same.

Thanks, and hopefully we'll get through this with as little pain as possible, with a glorious new era of something vaguely like modern net speeds and stability at the end of the process. Wish us luck.
Current Mood: hopeful

12th February 2011

10:55pm: Newmoon back up
Okay folks, newmoon is talking to the world again. Mail seems to be coming in and going out. Mailing lists are apparently working again, and it is now once more possible to ssh into the box.
It took solarbird getting home to help me beat the box into shape, though. And this is what we had to do:


  • Dara discovered the cd drive in the box wasn’t actually properly plugged in. Plugged that back in properly, only to discover that it was old and crotchety enough that it was refusing to read the rescue disc I’d created

  • Tried to burn a second rescue disc, but the old and crotchety drive wouldn’t read that either, so Dara swapped the old drive out for a different one we had

  • Using our backups from earlier in the week, I grabbed a screwed up copy of a file and got us back to the point of being to log in

  • Once we were able to log in, I had to reinstall a package called libssl, which I had to do by manually grabbing the .deb file since Debian’s actually had an OS update and apt-get wouldn’t let me get the version I wanted (for those of you for whom that actually makes sense)


Long story short, those of you who rely on newmoon, please, PLEASE check your stuff and make sure that everything looks okay. If you have files locally stored on the box please ssh in and check them. If you’re getting mail from newmoon, please doublecheck that your mail clients are behaving as you expect.

Please report any issues to Anna or Dara. Drop a comment on this post, or else send us email at either our murkworks or gmail addresses.
1:38pm: Newmoon damaged by power hit; murkworks mail services down
We just had a couple of power spikes at the Murk, like we do. This time though they appear to have taken out one of our UPSes--and it was the one that newmoon, our mail server, was plugged into. When I tried to bring newmoon back up, it worked for a while and then hung. I am not currently able to log into the system to restart its services.

So this means that as of this afternoon, mail services on murkworks.net are down. This impacts not only personal accounts but the various mailing lists we run. Please spread the word if you're on any of our mailing lists, and if you have backup personal email accounts, let people know they should use them this weekend if they need to get in touch with you.

solarbird or I will post here when we have more updates. Watch here, either of our journals, or my Twitter or Facebook accounts.

Sorry about this, folks. :/

- Anna
Current Mood: annoyed

25th October 2010

11:01am: Intermittent brownouts at Murkworks North
We're seeing intermittent brownouts this morning at Murkworks North; at the moment, they're just enough to cause lights flicker and WiFi interruptions locally. All the servers are, of course, on wired connections. If it stays at only this level, we'll have no problems staying online, but watch for the possibility of short-notice shutdowns in the event of power outage.

14th October 2010

1:52pm: Back online
Power restored around 12:40; all servers back online as of 13:45. Please resume normal... whatever it is you people do. XD
10:22am: Power out, murkworks north
Power has failed at Murkworks North. PSE reports equipment failure, and an estimated repair sometime after noon.
Current Mood: calm

23rd July 2010

3:10pm: Yes, network access is sucking
Verizon's having more hardware problems, or more likely, never really fixed the hardware problems giving us trouble earlier in the week. A new ticket has been opened with BIG ARROWS pointing to the "THIS PIECE OF HARDWARE IS FAILING" report.
Current Mood: frustrated

7th July 2010

11:58am: Heat-related shutdowns possible
While not nearly as badly off as most of North America, we're set to experience a record-setting heat wave here around the Nightmare Factory. While we're pretty good at heat management, we do not have full A/C, and the Murkworks may have to shut down some servers during peak heat times of the day. This will happen with relatively little notice. The order of shutdown will be, as necessary: backup services, web services, mail services, then DNS and complete network, in that order. Temperatures are expected to return to ranges more approximating normal this weekend.

Last year when we went through this we stayed up, except for backup services, throughout the entire heat wave, then had a power outage just as things cooled down. It was sad.
Current Mood: hot

19th May 2010

8:50pm: Back up - check for problems
We're back up but we've suffered a bit of damage. As far as I can tell all the servers came up fine. I haven't done much of a checkup yet but I'm working on it - we had a lot of issues getting power back up on all circuits and we've sparked a bunch of surge protectors, but it looks like we had them in all the right places.
4:57pm: Primary power failure, Murkworks North, immediate shutdown
After fluctuating violently several times, power is completely out here at Murkworks North. All servers are going offline IMMEDIATELY. No eta for return.

16th December 2009

1:56pm: Routing problems upstream of us
The murknet's upstream IP provider is apparently having some sort if difficulty; we can reach some local sites and some Of Vancouver, but most of the US is currently unreachable. We don't know why but it's too big a problem for them not to be aware, which is good since I'm not currently able to do much about it! Please carry on and hopefully it'll be patched up soon!

6th November 2009

5:24pm: A second night of severe weather
Hi, it's the figs of the Murkworks again, and we're having a second straight night of not just unusual but downright freakish severe weather - half a centimetre of hail, FOOM FOOM FOOM rain, and lots and lots of lighting, which we just don't see here. The cats are terrified. Fortunately there's not much wind associated with it so far, which makes it a lot less likely we'll lose power, but it's a possibility. Be advised that you may need to log out quickly on short notice.
Current Mood: excited

26th October 2009

8:32am: Winter storm advisory
We have our first winter storm advisory of the year, and with it, a series of brownouts that have hit what we internally call "Stage 2." (Stage 1: First alarms tripped. Stage 2: Multiple alarms, typically plus momentary darkness because we lose the lights. Stage 3: Extended power losses less than two minutes. After that, we shut down.) Hopefully it won't get any worse than this, but it could. Be prepared for power-related shutdowns on short notice.

30th July 2009

9:38pm: Power restored, Murkworks north
Primary servers back online. Backup server still offline for heat reasons.
6:52pm: POWER LOST, MURKWORKS NORTH: EMERGENCY SHUTDOWN
Power has failed at Murkworks North; we don't know why but it's not internal. All servers are being shut down immediately. No ETA for return.

29th July 2009

2:30pm: Server room at 30.1C
As per the previous update, we're dealing with record-smashing heat today, and Murkworks North is sadly not air-conditioned. Through a variety of tricks, we've been keeping the temperature down from 39.1°C/102.3°F outside, but the server room has just passed 30°, a bit after 2pm. We will begin shutting down services at 34°C. Be prepared for shutdown without specific notice. I'll be tracking temperature (of course) and will notify again at 32°C.
Current Mood: hot

27th July 2009

8:29pm: Heat-related shutdowns possible (if not likely)
We're experience a record-setting heat wave here around the Nightmare Factory, and while we're pretty good at heat management, we do not have full A/C, and the Murkworks may have to shut down some servers during peak heat times of the day. This will happen with relatively little notice. The order of shutdown will be, as necessary: backup services, mail services, web services, then DNS and complete network, in that order. We fully expect to take at least backup services offline Tuesday and Wednesday (28/29 July), with predicted highs around 37°C on Wednesday. It is likely that we will take down mail services as well on Wednesday, but we will work to avoid it.

Temperatures are expected to return to ranges more approximating normal on Thursday and Friday, or so we hope.

eta: We just broke 30° in the server room. Backup services are going offline now.

31st March 2009

11:37am: Unexpected winds
It's unexpectedly windy today and Murkworks North has experience a series of short power flickers substantial enough to trigger seven (and counting) Level 1 UPS alarms. (We have alarms that trigger at different degrees of interruption. Level 1 is highly transient and includes brownouts, Level 2 is substantial but less than one second, Level 3 is full power failure.) Accordingly, we may have to shut down servers with very little to no warning. Consider this your heads-up.

22nd February 2009

10:19pm: mail, web servers upgraded
lodestone and newmoon have both been upgraded with a minimum of fuss. so what the hell's the point of that? they've taken all the fun out of it with this whole shit-just-working thing. where's the joy of seething hate and unthinking, blinding rage? i miss it.
Powered by LiveJournal.com