February 12th, 2011

fig

Newmoon damaged by power hit; murkworks mail services down

We just had a couple of power spikes at the Murk, like we do. This time though they appear to have taken out one of our UPSes--and it was the one that newmoon, our mail server, was plugged into. When I tried to bring newmoon back up, it worked for a while and then hung. I am not currently able to log into the system to restart its services.

So this means that as of this afternoon, mail services on murkworks.net are down. This impacts not only personal accounts but the various mailing lists we run. Please spread the word if you're on any of our mailing lists, and if you have backup personal email accounts, let people know they should use them this weekend if they need to get in touch with you.

solarbird or I will post here when we have more updates. Watch here, either of our journals, or my Twitter or Facebook accounts.

Sorry about this, folks. :/

- Anna
fig

Newmoon back up

Okay folks, newmoon is talking to the world again. Mail seems to be coming in and going out. Mailing lists are apparently working again, and it is now once more possible to ssh into the box.
It took solarbird getting home to help me beat the box into shape, though. And this is what we had to do:


  • Dara discovered the cd drive in the box wasn’t actually properly plugged in. Plugged that back in properly, only to discover that it was old and crotchety enough that it was refusing to read the rescue disc I’d created

  • Tried to burn a second rescue disc, but the old and crotchety drive wouldn’t read that either, so Dara swapped the old drive out for a different one we had

  • Using our backups from earlier in the week, I grabbed a screwed up copy of a file and got us back to the point of being to log in

  • Once we were able to log in, I had to reinstall a package called libssl, which I had to do by manually grabbing the .deb file since Debian’s actually had an OS update and apt-get wouldn’t let me get the version I wanted (for those of you for whom that actually makes sense)


Long story short, those of you who rely on newmoon, please, PLEASE check your stuff and make sure that everything looks okay. If you have files locally stored on the box please ssh in and check them. If you’re getting mail from newmoon, please doublecheck that your mail clients are behaving as you expect.

Please report any issues to Anna or Dara. Drop a comment on this post, or else send us email at either our murkworks or gmail addresses.