I mean...

I have made
the first post
that that was available
on your blog

and which
you were probably
saving
for for something relevant.

Forgive me
it was exciting
so texual
and so gratifying.

If you do go with WordPress, I suggest BackupWordPress, which uses WP's cron mechanism to schedule daily and full backups, which it can then email to somewhere, anywhere.

And you're just in time to catch the first major version change in WP that makes the interface on the backend much friendlier.

And it checks for updates & stuff.

Now, of course, I have even less time to finish the paying work, but fortunately it's going well today. So maybe doom is not inevitable, yet.

So Don -- you got first post, but I'm pretty sure I had last post!

(It's weirdly ironic that the last post on ML before the disaster was on Clay Shirky's Here Comes Everybody, which is about the power of internet-enabled collaborative projects.)

But I'll volunteer to help with the mySQL coding, as needed.

Now Teresa and I are going to enjoy a large drink, and we commend the rest of you to the equivalent self-indulgence of your choice.

Regarding WordPress, some folks seem less enthused about that new interface. (via Amanda Baggs who got knocked offline for about a week by the upgrade.)

Perhaps an occasional Emergency is Good For Us, keeping the adrenal glands in good fettle and the various juices flowing, but there's also much to be said for keeping frequent and complete unflawed BackUps and living a more tranquil life. Indeed, this is something I should do with my own computer, Real Soon Now.

Best Wishes for getting a perfect restoration of the Making Light Archives.

(But seriously--having Making Light go missing felt something like the times our cats would go walkabout. I worried and kept waiting for the scratching at the door. Should I offer you some tuna?)

But speaking as someone who actually played around with the new interface instead of getting all crybaby upset when it was suddenly *different*, there were no bad changes.

Indeed, from a UI perspective, just about all the changes were good.

I think the folks complaining mostly hate change. I can do a lot more a lot quicker in the new UI, which is also faster and more limber. And if you're new to WordPress, there's no better time than now to start.

And yes---I f'in HATED WordPress a year ago and thought it was the suck. I have since learned differently. For instance, their plugin system has a better clue than most others out there I know of.

I don't know MT in detail or WordPress at all, but I am aware that MT 4 at least natively aggregates multiple blogs together, and may well have other features that you all are currently kludging together by hand. If the stability problem has to do with the complexity of your MT install, it's possible that an upgrade to MT 4 might be easier than converting to WordPress.

Perhaps BoingBoing's sysadmin could give you some advice? If they're running even a modified version of something available out of the box, that would seem like a good place to start for a stable system.

(I know that ScienceBlogs runs on MT, and they have quite a lot of traffic. OTOH they also have, you know, staff.)

It was rather like that piece of hate-mail making its rounds of the writing/agenting/publishing world.

But I'd be happy with anything you guys use as long as you didn't have to deal with complexity and the backups could be automated. Anything else is gravy, with WP or without it.

For instance, I believe they're set up to serve static pages due to their heavy load. I'm sure their experiences are instructive, but I don't know that they'd be directly applicable.

Seriously, it's nice to see ML back in something like its normal form, even with a bit of hopefully temporary amnesia.

My only gateway onto MakingLight is very expensive, and I miss many important postings ...

Out of the box, it's supposed to be a CPU hog, for even moderately popular sites. There are at least three decent plugins for caching that work pretty darn well--Scalzi uses one for Whatever. Jeff Atwood, of Coding Horror, had an excellent post titled Behold WordPress, Destroyer of CPUs that explores the problem in some depth. The comments there, as with MakingLight are instructive, though sadly lacking poetry and humor.

Especially worthy of note is an addendum to the article referring to Matt Mullenweg's MySQL tuning tweaks and a reference to a Google-Summer-Of-Code project designed to add caching to WP.

While I have enjoyed using WP and it's plugin ecosystem in the past, I am not a platform fanboy, and whatever you decide to do won't materially affect my ego ;)

John A Arkansawyer @ #21, I was thinking about a long-term solution, not recovery.

So much thanks to all who have all lit (or held up) candles and made light rather than cursing the darkness.

Thanks specially from me to Epacris for checking on my stuff. I'm not in the best of states for organising or keeping track of things. Soon will need to lie down for a while after being up and on the 'pooter.

And thanks in advance to those who'll do it for all the work that will be done to restore things to a happy future equilibrium.

Actually, though, I am really glad you guys got it all sorted out. Also, let me add my regrets that I was completely useless in the crisis.
Sorry about that.

I think the poster is mad, and if any mail with their address ever comes again to that agent (or whatever) I'm guessing it will get returned as a turn-around mail. Like, who needs that kind of grief.

For that matter, what would happen if they got published but bad reviews? Double yikes.

IF I get something written, polished, etc until I think it's ready to go out, out it goes, I'm on to the next thing and there you are. I'm surely not going to complain about rejection that's just part of the job. And I had Marion Zimmer Bradley as an editor, so I got used to being thick-skinned about comments...

Just saying.

I just sent the March Sidelights & Particles. Took me three tries to get the ones with live links. Sigh.

Glad to see the Light again. G'night,
joy

1. Visited Making Light

2. Read first post on top, i.e. Open thread 102, i.e. Abi's delightful mashup involving brain-eating zombies and our hostess's Voice

3. Wondered at the feeling of déjà vu

4. Realized, it's Making Light, déjà vu and other paranormal experiences are par for the course

Made sense to me!

In other news, I just finished Little Brother, and it's awfully fine. I think I'll loan it to someone at work tomorrow--maybe our network security officer. That might be fun.

No, wait a minute. The wife gets first crack at it. What was I thinking?

Anyway, I have three things to say, any of which might be spoilers, so:

Cntr 256 jnf gur bayl cynpr V gubhtug V jnf tbvat gb pel.

V'z fb tynq V jnf jebat gung Znfun jnf Qneely. He set that possibility up at one point.

Jura bar bs gur svefg unpxf va lbhe obbx vf yvsgrq sebz Qbhoyr Fgne naq Oehpr Fpuarvre gura ersreraprf vg, lbh ernyyl bhtug gb zragvba gur byq zna.

Oh, and my laptop seems to be cooling off. False alarm. I think.

Is there anyone whose posts are known still to be missing who I should look for?

Y'all rock. Now, please excuse me while I back up my laptop...

Good work reconstructing things thus far, everyone. Sorry I wasn't around to help, although I bet lots of you are quicker and better at it than I would have been.

If thoughts of a full backup cause stress, then perhaps start with a minimal backup, just for practice.

For example, take a 128MB compact flash card and a 128MB USB thumb drive. Both should cost just a couple of bucks at most**.

With the CF card and a (borrowed or owned) digital camera, spend 10-20 minutes to go around your place and take pictures of important things--all rooms, clothes in the closet, the book collection, important documents. One good rule for photos for insurance--have the camera on a steady surface, not handheld.

With the thumb drive, spend another 10 minutues copying important files and the thumbnail pictures of your digital photo collection onto it. Drag and dropping is fairly fast w/ USB. Include an export of your bookmark files.

Take both memory devices and bring them to a trusted someone's house to be your interim backup until you can do a more complete one.

-----------
* something like a Backup Amnesty Day

** considering that 4GB+ is $10, 128MB and under seems like it should almost be free... i.e. small drives forgotten about in the back of drawers.

May your nerves now have time to become unjangled.

(Also, I've taken this opportunity to disguise my email address. My previous "view all by" email address doesn't include the capitalized bit.)

I was thinking about the best way to get the comments back, without having to do an elaborate reconciliation of "who's sent in which post with whose emails as of which date", which seems kind of crazy-making just from looking at the spreadsheet.

I saw up there aways in some of the posts on abi's site that somebody's already written a Perl script to parse the HTML for either a thread page or the "view all by" pages out into the individual ocmments, and get the relevant fields out. (Thread name, poster name, URL, email, date/time.) Ok, great. Then you also have a ton of overlapping sources for various versions of the posts.

I think what I'd go with at this point is to invent a simple rule for munging all the above into one key, which should uniquely identify a specific post. Use that as a Unix filename (or possibly subdir + filename) per post.

Tweak the Perl scripts to take input of either a view-all-by page or thread page, skip any cache header, digest it into the posts, and spit each post out into a unique file named by the unique key. If the file already exists, just skip it! Wherever it came from, it should be the same content modulo whitespace. (If you want to get real fancy, if it does exist compare the new version to the old version, and write something to an error file if they differ.) Special case: if the file appears to be truncated, don't save the last thing in it, as that might be corrupt.

Now on your Unix box, just pour every HTML file TNH & PNH have been mailed into this script. Don't worry about dups or which ends latest or which is the best source, just shovel it all in. At the end, you should have all the individual posts the hivemind has collectively been able to recover, each in its own file, and keyed on its critical attributes; now you can just write one last quick script to dump all the resulting files into the database.

This trades off some waste of CPU cycles to avoid wasting a lot more brain cycles, which is the thing to do. (Also should be way cheaper to do this reconciliation of dups in the file system rather than the database.)

Especially if you drop the resolution down just a bit, you can fit 100 or 200 pictures onto a CF card that costs less than one roll of film (and/or 1 roll processed).

Why not use that cheap CF device for backup or a preliminary backup--it's a nice gift to your future self to take care of that now.

Oh well, back to the cooking.

not ur cheezeburger

As for Patrick's comment that
Our server fell over hard. Smoke came out,
that brings up fond memories.
Not.

Luckily, there were plenty who stepped into the breach.

Congratulations, all of you.

Lemme know if there is any grunt-work that needs doing. Happy to help.

Death to vermin.

I'm running several different WP blogs (albeit low-volume ones), including my URL one. I have a bunch of experience with PHP, Web programming, programming in general and computers in general, including the Stockholm University Latin 101 webcourse (http://primalatina.klassiska.su.se), and I really, really want Making Light to stay around and thrive.

This was entropy on a Harley V-Twin with bored cylinders, running on nitro. There's nothing that looks less like information than the contents of a dead hard disk after the fire's been put out. That there are people who make a living resurrecting the contents of disks after immolation is a tribute to the ability of humans to learn how to ride such beasts.

I started running Time Machine on my laptop a month or so ago, archiving to a hard disk attached to my wireless base. It was originally supposed to be a RAID 500 Gig mirror, but it turns out that the Airport Extreme can't see a RAID disk for some reason or other, so I fell back to a single 500. Time Machine has a great UI to the restoration of files; it's basically the Finder with a big slider to set the date and time you're looking at. And getting an incremental backup every hour automatically, and a graceful termination if I walk out of the house with the computer during the backup is just great.

But, of course, it's when things fall over dead that you find out all the things that aren't right. When I saw the news about ML, I went looking for my Firefox cache files (my tabs were clear; I'd recently restarted Firefox because it gets into memory suck mode if left up with a lot of tabs for more than a week). Guess what? No cache files at all! Firefox got into some sort of mode where it was only caching in memory, even though disk was enabled and I'd allowed 200 Mb of disk cache. Still trying to figure out what's going on; about:cache doesn't even list memory as a possibility.

I'm impressed with how quickly it all came back here; a lot of us worked very hard on this. Now that success is in sight (I'm looking at it), I'll relate a much worse experience. Last January I was put back on the project I had been hired to work on a year and a half before: an internal webapp for manufacturing in a musical instrument company (a subsidiary of Roland that makes pipe and church organs). It had been running in production* for year and everyone figured it was all good. Famous last words. A few weeks later the database lost about 90% of its contents, so I asked the sysadmin responsible for databases to restore from backup. His reply: "I don't do backups for that system." "So who does?" "No one." For a year.

It was MySQL, but it hadn't been setup properly, and wasn't storing to binary log files, so the internal files on the disk were toast. I spent three days trying to reconstruct the data, but couldn't get more than about 20% of it; the rest had been overwritten during the crash. Needless to say we now do backups every night, and we test them every month. Where'd the data come from? It's a relatively small database, so it was possible to re-enter everything manually. Not that it was fun.

* Nah, it was a really kludged-together prototype that was pressed into service when management decided they didn't want to put any R&D money into it for a year or so.

Sympathies, and congratulations to all whose work has restored the site.

As someone who has suffered two laptop hard drive "sudden deaths", I already back up my on-going work on a daily basis, and periodically do large backups. I will reiterate what others (e.g. Kathryn from Sunnyvale @ 58 ) have already said: backup, backup, backup.

Remember that if thieves come in they will probably grab your backup external hard drives as well as your laptop (and both will get ruined in event of a fire), so an additional set of backups on DVD, kept at another location and kept up to date, is a good idea.

I just wish it had been possible to keep a "backup" of my folding bicycle - which got stolen last week.

This is a very good point. I think that having a second external drive that you keep at [work|mom's|friend's] that you swap for the one at your desk is more effective. Data on harddrives like to be refreshed and verified by reads*. Data on DVDs is harder (more time consuming) to verify manually.**

*apparently this is handled in the background by the harddrive controller and modern file systems like NTFS and HFS. Who knew?‡
**there are some programs that create parity data that can automagically check to see if these are the droids files you are looking for, but I don't understand them or how to configure and use them. I do understand plugging in external harddrives.
‡ I'm amenable to correction here. Bruce, or Greg, what do you think?

With server shadowed, thought darkened in night
a click repeated, page refreshed once more
hope returns, the minds of the web make light
the work of hours finds a memory's core
And here we give thanks for our gracious hosts
and the host of others who gave of time,
life, who searched for hours so that we have most
of the pithy commments and lolcatz rhymes
Again we refresh and this time are bless'd
to see again the hushed murmur of crowds.
I am making a note here, huge success
A working of code of which to be proud
The haven returns, the light is restored
celebrate day and the net overlords!

please excuse the first attempt-- there's a reason I don't try and play this game, but it seemed appropriate.

Teresa has a similar method, which is why, in the order of the last week's various crises, the fact that her work-provided MacBook dropped dead on Monday barely made third place. We unplugged the Time Machine drive and had her up and running in less than an hour with a completely restored system in her own account on the desktop Mac. The next day, we dropped the afflicted notebook off at Tekserve and rented a temporary replacement; the only real delay lay in figuring out that if you're restoring from a Time Machine backup to a machine that originally ran Tiger, use Migration Assistant rather than booting from the Leopard system disk. Once we got that sorted, everything worked pretty much perfectly.

So, yes, we're pretty backup-conscious with our personal data, which makes it even more inexcusable that we let our Making Light backup strategies fray so much.

Contact info is the lj name in the link, at juno.com

Susan, email Tracie, Emily or I please?

Fortunately, we shouldn't be missing too many comments to start with, and those we are missing are at the ends of threads; otherwise, our @nn system for followups would be pretty beat up.

In re general backup-ing -- I'm crap with backing up. I really am. Every few years it bites me, but I regard it as a sort of diet for the soul. Pining for the missing data makes it far more poignant than any amount of actual data could have. And then occasionally I'll find something on some forgotten disk -- nowadays that usually means building specialized equipment to read it -- and there is much rejoicing.

Appropos of such- Please cue pre-k'zoo panic.

I didn't used to be very good at backing up. The first sudden, catastrophic, hard drive failure encouraged better backing up. The second one (oh, that dreadful gr-gr-grinding sound of hard drive reducing your data to bits (pun intended)) less than six months later really made me pretty paranoid. Nowadays, if I've had a really productive morning, I'll save to an external drive then, not wait for my regular evening backup.

If we had time-to-the-second, as it goes into the database, yes. However, AFAICT, only the nearest minute ends up visible in the HTML, and occasionally people do post twice in the same minute. (Usually the second is something like "Oh hell, I know how to spell that word correctly" but occasionally it's more than that.) I don't know if keying on just poster+minute would lose anything from the last two months, but it's not impossible.

I just realized, though, that the HTML anchors such as 010011.html#253523 should work OK for a unique working key, and they appear both in the thread and author views.

I sent a note to Patrick & Teresa, but I'll offer to you too: let me know if you need more Perl hackery. I know nothing about MT, but I've written teeny scripts to large systems in Perl.

As it was set up using FreeBSD's gmirror on the drive level, I lost zero data. It took me a little while to figure out how to go back to running it from one drive while I ordered a new drive to rebuild the mirror because I had not done some of the setup ideally, but I got it up again, and then remirrored it to the fresh drive. No data loss whatsoever, despite an unreadable HD.

Drives are cheap now; using two in place of one is better than a backup strategy. Despite this, backups are great against software failures, where you (or your computer) does something which corrupts or wipes out all your data.

And sadly I couldn't resist the urge to reply.

I am, however gladdened, amazed and awestruck at how much this piece of the inellectual landscape means to so many of us, and how much talent and expertise could be; and was, brought to bear.

I'm sorry that I didn't have much to offer in the recovery.

I'm writing this on a Lenovo 3000 c100, now out of print. Good touchpad, good 'mouse' buttons (though too loud for use in bed with a sleeping spouse, which is not a problem unless you need to right click.) Matte finish non-widescreen, enough single cpu/core power to play divx/youtube at standard def. Perfectly adequate, and comes with only a handful of easily uninstallable trial versions. (Excepting for a full version of WordPerfect, which is easily installable, but not a trial version at all.)

OTOH, I would recommend getting a Dell with Ubuntu, and then running winXP in a virtual machine. [infodump warning] As you know Bob Serge, a virutal machine is a snapshot of your computer stored as a file on a harddrive, making it possible to have an easily archived copy of your current windows install that you can recover to when the latest update/malware/exploit hoses your operating system. (A "small" XP install takes about 6 gigs, so you can't save your machine's state every couple of minutes - at least for a few more years.)

Why would that be useful to a writer? 1. It would enable you to keep your work "computer" isolated from your "need to install this buggy application" computer. 2. More importantly, if you get xp set up just the way you want it, you can easily migrate it to another box just by copying a file. There's nothing like working with a completely familiar environment when you are writing. 3.Long after the world has moved on to using hyperintelligent space jellies interfaced with the medulla oblongata for word processing, you can still run WordPerfect 5.1 in a dos box if you want.* More realistically, you can drag that virtual machine file onto, say, a MacBook**, or another computer running XP if you want to switch platforms. [end infodump]

I don't have anything meaningful to say about the ASUS Eeepc or MacBooks that hasn't been said better, elsewhere. By all that's holy, I'd love either one, and I want to believe that the Eee would be <blink> the perfect </blink> laptop for a writer, but I have no empirical evidence that this is the case.

*the adventurous could run a VM of a DEC quad alpha with a vt-220 terminal simulator to run WordPerfect 5.1
**or iPhone mark IV.

(Crossposted in the Buckley thread that has highlighted this issue.)

Reading the threads at abi's site was an experience, initially gut-wrenching (I got that sinking stomach feeling on reading of the (potential) loss of the last two months) but eventually, triumphant.

Bravo!

Mez, 34,
If only it had been better poetry ;)

Clifton Royston, 94,
RAID 1 is great and all, but it doesn't protect you against user error, particularly ones where bad data is faithfully copied to the second disk.
[are you sure you want to delete]
[NO] I want to delete
[YES] I want to delete
[CANCEL] I don't want to save anything before shutdown
It also doesn't protect against power supply failures that can take both disks with it.

At the risk of continuing to be a Forth Yorkshireman, a cheap UPS unit* does usefully remove one of the single points of failure: dirty or non-existent power reaching those precious precious disks. Handy, when your flatmate's antique air conditioner dims the lights. (An hourly occurrence, now that spring has arrived.)

*47$-63$

Anyway, I gnashed my teeth and wailed and I'm glad I can stop now.

As regards Patrick's
Our server fell over hard. Smoke came out

Alas, once let out, the magic smoke cannot be put back in.

Is this something that is fixed by the upgrade?

Oddly, it seems to me that I get fewer errors with Firefox than Safari. I've been wondering if Safari caches the content of forms and is interacting badly with wordpress, although that doesn't make sense to me.

My blog is linked from my name if you are curious, although it is mostly aimed at other mathematicians.

GUI interface! Like ATM machine?

Sorry, just crowing at acronym drift. (This isn't a jab: language changes, acronyms get de-acronymized, and I can't help but feel wonder at the birth of a new error/non-error usage chimaera.)

Unfortunately, the GUI seems to mess up worse than the HTML on mine, which is why I have to edit it manually in the first place.

I seem to have less trouble with it on Internet Explorer than with Firefox, usually. Different computers create different line break issues, as I discovered on my vacation -- one internet cafe made every line break show up as triple or quadruple, my home computer usually makes it show up as a single carriage return, and ... etcetera.

I think I will try that upgrade. It can't make things worse at least.

Don't say that! Uh oh, too late... now you've done it! :-)

If I had to distill ten years of experience down to a few points -

1. DO the damn backups.

2. Check that the damn backups actually ran. Check what was backed up.

3. Test your damn backups. This doesn't mean restoring an eensy-meensy .txt file. This means restoring something large - comparable to your main database. Do this once a month.

4. Store the most recent damn backups off-site. Yes, it's inconvenient when the server crashes, but much better that than losing both your main drive and your backup if your house burns.

5. Have more than one damn set of backups if at all possible. At least, backup up your work files twice - your unfinished novel, the family photos, etc.

6. Label your damn backups so that you (or someone) can tell what's on them without a computer. Feeding 32 discs/tapes/whatever into a drive so you can find your latest backup is not fun.

I escaped that job nearly a decade ago, and I haven't kept up with the latest tech - but I have no doubts whatsoever that the principals remain the same.

Thanks and sympathies to Patrick and everyone who helped put Humpty Dumpty together again - it's an ugly bit of work to have land in your lap.

In the rare case that WordPress/Markdown misbehaves, I translate everything to HTML outside the WordPress interface (I write most blog posts in a text editor which can run the Markdown script, so this is simple for me) and strip out all the line breaks (also simple, since the text editor also has a lot of text manipulation options).

(Sounds like another HTML for Dummies post though: fixing the spacing in your d*man Blogger|Wordpress|Typepad|MoveableType|LiveJournal theme.)

Anyways, the new WordPress's GUI editor has been upgraded and is mostly sane. But having it turned on in prior WordPress's did tend to screw up HTML, even if you weren't using it. Gods know why.