Before the Shadow Play

Category: Personal

  • Leveraging an outage to build community and consensus

    We had our first extended outage at RadioSpiral this weekend, and I’m writing about it here to point out how a production incident can help bring a team together not only technically, but as a group.

    The timeline

    On Sunday evening, about an hour before Tony Gerber’s Sunday show, RadioSpiral went offline. Normally, the AirTime installation handles playing tracks from the station library when there’s no show, and it played a track…and then stopped. Dead air.

    The station has been growing; we’ve added two new DJs, doubling the number of folks who are familiar with servers, Linux, etc. Everyone who was available (pretty much everyone but our primary sysadmin, who set everything up and who is in the UK) jumped in to try to see what was up. We were able to log in to AirTime and see that it was offline, but not why; we tried restarting the streaming service, and the server itself, but couldn’t get back online.

    We did figure out that we could connect to the master streaming port so that Tony could do his show, but after that, we were off the air for almost 12 hours, until our primary sysadmin was up, awake, and had finished his work day.

    A couple hours of investigation on his part did finally determine that LetsEncrypt had added a RewriteRule to the Airtime configuration that forced all URLs to HTTPS; unfortunately it needs HTTP for its internal APIs and that switchover broke it. Removing the rule and restarting the server got us back on line, and our very patient and faithful listeners trickled back in over the day.

    Now what?

    While we’d not been able to diagnose and fix the problem, we had been chatting in the staff channel on the RadioSpiral Discord server, and considering the larger issues.

    RadioSpiral is expected to be up 24/7, but we’re really running it more like a hobby than a business. This is reasonable, because it’s definitely not making any of us money, at least not directly. (Things like sales of albums by our DJs, etc., are their business and not part of the station’s remit.) This means that we can have situations like this one, where the station could be offline for an extended amount of time without recourse.

    Secondarily, RadioSpiral is small. We have three folks who are the core of actual station operations, and their contributions are very much siloed. If something should happen to any one of the three of us, it would currently be a scramble to replace them and could possibly end up with an extended loss of that function, whether broadcast operations, the website, or community outreach and the app.

    So we started looking at this situation, and figuring out who currently owned what, and how we could start fixing the single points of failure:

    • Station operations are on an ancient Linux release
    • We’re running an unsupported and unmaintained version of Airtime. It can’t even properly reset passwords, a major problem in an outage if someone can’t get in.
    • The MacOS/iOS app is handled by one developer; if that person becomes unavailable, the app could end up deleted from the store if it’s not maintained.
    • The website is being managed by one person, and that person becomes unavailable…well, the site will probably be fine until the next time the hosting bill isn’t paid, but if there were any issues, we’d be SOL.
    • We do have documentation, but we don’t have playbooks or process for problem solving.
    • We don’t have anywhere that is a gathering point when there’s a problem.
    • We don’t have project tracking so we can know who’s doing what, who their backup or backups are, and where things are in process.
    • We don’t have an easily-maintained central repository of documentation.

    What we’re doing

    I took point on starting to get this all organized. Fixing all of the things above is going to take time and some sustained effort to accomplish, and we’re going to want to make sure that we have everything properly set up so that we minimize the number of failure points. Having everyone onboard is critical.

    • We’re going to move operations to a newer, faster, and cheaper server running a current LTS Ubuntu. [Done.]
    • We’re going to upgrade from the old unsupported AirTime to the community-supported LibreTime. {We did better, and moved to Azuracast.]
    • We’re figuring out who could get up to speed on MacOS/iOS development and be ready to take over the app if something should happen that I couldn’t continue maintaining it. At the moment, we’re looking at setting up a process to bump the build number, rebuild with the most current Xcode, and re-release every six months or so to keep the app refreshed. Long-term we’ll need a second developer (at least) who can build and release the app, and hopefully maintain it. [There’s enough active development happening that the going idle isn’t a problem, but the second dev is still bus factor 1.]
    • We haven’t yet discussed what to do about the website; it is already a managed WordPress installation, so it should be possible to add one or more additional maintainers. [Rebekkah is still primary, but we can all get in and do things now.]
    • We are going to need to collect the docs we have somewhere that they can be maintained more easily. This could be in a shared set of Google docs, or a wiki; we’re currently leaning toward a wiki. [Wiki up on the main site.]
    • We need project tracking; there’s no need for a full-up ticketing process, at least yet. We think that Trello should do well enough for us. [We added a ticket system inside the main site; working okay so far.]

    We have set up some new Discord channels to keep this conversation open: #production-incidents, to make tracking any new problems easier, and #the-great-migration, to keep channels open as we move forward in the migration to our new setup.

    Everyone is on board and enthusiastic about getting things in better shape, which is the best one could want. It looks good for RadioSpiral’s future. Admittedly we should have done this before a failure, but we’re getting it in gear, and that’s better than ignoring it!

  • “Projects in Flight”

    First a confession. I tend to have enthusiasms, work hard on them for a while, and then have something else interesting come across my radar, which will then become my new enthusiasm. This tends to lead to a lot of half-completed things, which I then feel bad about and avoid, causing me to not get anything done, making me feel even worse.

    I’ve decided that I’m going to try a different strategy: “projects in flight”. I’m embracing the fact that I have enthusiasms, and lots of them. I contain multitudes. And this is good.

    So instead of feeling bad that I have a dozen projects that aren’t getting anywhere, I’m going to acknowledge that I have a lot of interests, and more of them than I have time to do. So some of them don’t pan out. Some of them get partway through, and then I discover that the problem is better solved a different way, or that the thing I want to do isn’t actually as good as I thought, or whatever. I am allowed to fail.

    Think about it this way: for every Google or Facebook, there are a hundred startups that try to do something, get partway in, and fail. Maybe the idea wasn’t so great. Maybe the resources to do the thing they wanted to do just aren’t feasible, or available, or affordable. Maybe they just can’t get someone to give them the seed money to try.

    All these projects fail. And the entrepreneurs don’t feel bad about themselves if they do. They gave it the shot they could give it, with the effort and resources they had at hand, and it didn’t work out – and they move on to their next project.

    So I’ve decided to embrace the entrepreneurial mindset for my personal projects. I’m keeping a list of everything I’m doing, from the trivial to the complex, and allowing myself to be happy that I am creative and multifaceted; if something doesn’t get done, it stays on the list as something to come back to, unless I decide it’s not worth coming back to…and then it goes into the “idea pool”. Maybe it’ll trigger something else later. Maybe it won’t. It’s fine.

    It hasn’t failed. I haven’t failed. I’ve just discovered something that as I approached it this time, it didn’t succeed. It was my AltaVista, or Ask Jeeves, or Yahoo! Search instead of my Google. Maybe on another look later, with more information, more experience, more time, more energy it will succeed.

    But I don’t have to feel bad about it anymore. I can be proud and happy that I’m trying things and doing things. Yes, I do want to finish things too, but I can stop looking at the unfinished things and thinking that I’m failing because they’re not all done and perfect.

    So: I have a dozen or so projects in flight, at various levels of done, and I’m happy that I have interesting things to do!

  • Archiving papers: a strategy

    I’m helping a friend archive a lot of notebooks and papers that they’ve accumulated over several years of writing. They’d like to be able to travel, but are a little worried that not having any backup for all this work is risky; fires, floods, and theft do happen, so even a fireproof box isn’t a guaranteed backup.

    We’ve therefore been photographing the papers, page by page, and creating a 3-2-1 backup of all of the digital photos. After some experimentation, we’ve come up with a workflow that works very well:

    • Create a Photos library that is not the primary. (She has an art business and needs to be able to use her iCloud-synced Photos library without it getting cluttered up with hundreds of photographs of pages.) This is most easily done by holding down Option and launching Photos. When the “select the library” dialog comes up, create a new one.
    • Photograph the items on a second iCloud account’s primary Photos library. This automatically syncs them to that accounts iCloud Photos.
    • On the machine where the secondary Photos library lives, log into iCloud.com with the second account.
    • On that same machine, open Photos with the non-primary library. (Hold down the option key and open Photos to allow Photos to select the non-primary Photos library.)
    • As batches of photos are taken, wait for them to sync to iCloud, then on the iCloud.com page for the second account, download the batch to the machine where the secondary library lives.
    • Create a new album in that secondary library, and drag the new batch of photos into it.
    • Put a sticker on the notebook/folder, and write in an ID (A, B, C, etc.) and the date it was photographed last. This allows active notebooks to be archived safely. (You should also add a note on the last page scanned with the date and album ID so you can cross-check.)

    Photographing the cover of the notebook/the file folder the pages are in helps make sure that you keep different batches of photos separate. If you do this, it’s much easier to keep track of which pages belong in which album, and gives a better way to track back which things are done and which aren’t.

  • iTunes Swedish Death Cleaning

    If you haven’t heard of “Swedish Death Cleaning”, the idea is that when you finally do drop dead, it’d be polite to not saddle whoever is taking care of your stuff with a big job of “is this important? should I keep it? should I just give all this away, or throw it away, because it’s just too much?”. Also, living with just the stuff that actually means something to you on a daily basis, as opposed to “I may want this someday, so I’ll keep it in my live gathering dust and generating clutter.”

    I definitely need to do more of that in my physical life, but this weekend I embarked on it in my digital one. Like most people, when I finally had iTunes and no longer had an actually “how full are my shelves?” physical limit, I started hoarding music. I had a lot of stuff from my old CD collection, music I’d bought from iTunes, the StillStream library from when I was maintaining the music library for that station’s ambient robot, music from friends who’d lent me CDs, stuff I’d borrowed from the library and timeshifted into iTunes to listen to “later”, free releases from Amazon…basically a huge pile of stuff. Worse, I’d put all this in iTunes Match, so even if I cleaned out my library, turning iTunes Match on again would just put all the crud back.

    In addition, my partner didn’t have a music library at all because her internal disk on her laptop was too small to keep all of her work and optional stuff as well. There was an offline copy of her old music library, and it too had also grown over the years from music lent to her, music I thought she might like, etc. She wanted to be able to pack up her CD collection and put it into storage, and maybe get rid of some of it as well. So we needed to take our old libraries and clean out anything that we didn’t want, and then see what each other might have that the other person might want afterward.

    I spent a couple evenings last week ripping the CDs she didn’t have online yet into a separate library, so they wouldn’t be part of the existing mess, and then went through and did the following in a brand new library:

    • Anything she actually owned got copied in. iPhoto’s ability to let me photograph the discs on the shelf and copy the text off of them came in very handy to make sure i got them all.
    • Anything I didn’t find in the library on that pass got ripped into this new library.
    • The not-previously ripped CDs in the secondary library were copied in.

    At this point, she had a clean “definitely mine” library. Now it was time to clean mine up. I had done one pass already to strip it down, but I wanted to make sure that I both cleaned out my iTunes Match library and made a conscious decision, “keep or not” for anything in there that I didn’t already have in the stripped-down library.

    The easiest way to do this was to to create a brand new, empty library, and connect that to iTunes Match, after turning on the “I want lossless copies” option — this is apparently new in Ventura, and is very welcome. Once this synced up, I could download and copy in only things I knew I wanted to keep. This meant I would actually have to look at the music and say, “do I really want to listen to this again?”, but not having to pull it out of an existing library would help.

    In addition, my partner had asked me to give her a copy of music of mine that I know she likes; we share a liking for world music, and several different other artists. After a little thought, I came up with the following:

    • There’s probably music in iTunes Match that we both want, and there’s definitely music I want. So let’s do this:
      • Create a new folder on a scratch disk that will contain music to add to her library.
      • Do the same for music I want to add to mine.
      • Drag those into the favorites in the finder.
      • Drag the Media folder from my target library to the sidebar as well. This will let me quickly check to see if a given release is already in my library , and if it is I can skip downloading it altogether, unless I want to give my partner a copy.
      • As I process each release in the Match library, I do the following:
        • If my partner would like it, download it.
        • If I want to keep it myself, open a Finder window using the Media folder shortcut and check if I have it.
          • If I do, simply delete it from the iTunes Match library (which also takes it out of iTunes Match).
          • If I don’t, download it.
        • If I downloaded it, right-click on one track in the iTunes list, and “Show in Finder”. This pops up a new Finder window with all the tracks for the release in it.
        • Command-Click on the folder name in the top bar of the window and go up one level to see the release in its enclosing folder.
        • Drag the release folder to the sidebar aliases for the “music to add” folders as appropriate.
        • Delete the tracks in iTunes. This removes them from the iTunes Match library, and iTunes Match as well.

    This took the better part of two days to finish, but I now have two cleaned-up music libraries, and an empty iTunes Match. I am considering whether to retain iTunes Match, mostly because it’s not a “backup” — it’s just a convenient way to share music across my devices, and doesn’t guarantee I’ll get the original file back.

    I’ve probably lost fidelity on some of the tracks I added to Match, and it’s possible some of them now have DRM. I will do another pass at some point and see; I’m not sure if it really makes a lot of difference to me right now, but I can always play them through Audio Hijack and re-record them to remove the DRM if I decide I want to.

    We also wanted a list of “what you have that I don’t” for both the final libraries; I was able to do that with Google Sheets, but I’ll post that as a separate article.

  • Useful shortcut for cleaning up files

    Useful shortcut for cleaning up files

    The situation

    I’m in the process of moving from one computer to another. My old 2010 MacBook Pro is still running very well with a replacement SSD for its internal disk, but it’s stuck at Catalina and won’t be going any further, mostly because the firmware has a password which I’ve lost, and Apple can no longer unlock machines that old.

    So if I want to do development in a recent Xcode, and I very much do, I need to upgrade. One side-effect of my recent layoff from ZipRecruiter was that they let me keep my machine, so I now have a 2021 M1 Pro that will run Ventura. (It’s possible that I’ll never need another machine, given that Apple machines stay supported for ~7 years; in seven years I’ll be 73, and either dead or unlikely to be programming on a daily basis.)

    The problem here, though, is that the internal disks are considerably different sizes. The old machine’s internal disk was 2TB, because that was the biggest affordable SSD I could get at the time. The new machine’s disk is 0.5 TB, and a straight copy from the old machine to the new is not an option — the immutable law of storage is that if you have it, it fills up — so I need to clean up the stuff I’ve got on the internal and move it elsewhere.

    I’m using a mixed strategy for this:

    • Anything on the internal disk will be there because it has to be.
    • Anything I want to keep and be able to access, but that doesn’t need to be available right now is going on Dropbox. (I will have to back this up separately; I’m going to work out a script to back it up with Backblaze.)
    • Anything that I need quick access to will go on an external 2TB SSD, which I will back up with Backblaze.

    So far, I’ve done the following:

    1. Gotten a copy of my most recent backup of the 2 TB internal disk from Backblaze on a 4TB spinny disk. (Costs me the price of the spinny disk, but worth it.)
    2. Copied the failing spinny disk copy of my old backups to an external SSD. (In hindsight, it should have gone to the empty space on the spinny external; I may do that later).
    3. Started walking through the SSD copy of the old files to clear space on the SSD for the files I want from the Backblaze spinny disk.

    The actual meat of this post

    So fine, I’m cleaning up the SSD. The actual thing I want to note here is that I have a collection of ebooks on that external that I want to file onto a folder in Dropbox. Problem is that a lot of them are probably already there, and the drag, get the duplicate dialogue, dismiss it, trash the file process is tiresome on the hands. I discovered a significantly faster way, and I’m noting it here for anyone else who might be doing something similar.

    1. Open the source folder (for me, that’s the “books” folder on the SSD) and the destination (that’s a categorized and subfoldered “Books” folder on Dropbox).
    2. For each file in the source folder, use the Finder search field in the Dropbox window, limiting the search to just the “Books” folder on Dropbox, and start entering the name of the source book.
    3. If the book is there on Dropbox, you’ll find it — and if there a duplicates, you can clean up the duplicates right from the search results.
    4. If it’s not there then it can be dragged over to the appropriate folder in “Books” on Dropbox after clearing the search field.
    5. In either case, the book is now either found or filed, and can be removed from the source folder.

    This is way faster and easier on the hands than dragging and dropping the books one at a time.

  • Considering the Cloud

    After the LastPass revelations and reading Jason Scott’s FUCK THE CLOUD essay today, I started considering what I should be looking at in terms of data security this year.

    Not as “can this data be stolen”, but as “can this data be lost irretrievably — and how bad would it be if it was?”.

    I have already lost access to my Twitter account, but I don’t think there’s much there that I’d care about if I never saw it again.

    I still have the EMUSIC-L archives, even though the ibiblio site has been broken for years. They are incomplete; we lost some of the really good stuff, including Mike’s hot-off-the-experience posts about the first Team Metlay gathering. Still, okay.

    My VFXsd sequences and patches are backed up on slowly-deteriorating diskettes, and it’s only a matter of time before those go. I think I have sysex dumps of all of them; I can replace the diskette drive with a USB one, but the SD-1 is getting long in the tooth, and I’m not sure I really mind if the various didn’t-quite-ever-amount-to-anything sequences are lost before I record them.

    Photos. I have several dozen photo libraries in various states of cleaned-upness, and that is a project I should devote some time to actually catching up on, even if it’s simply to pull out the good ones and let whatever happens to the rest, happen.

    Facebook does allow you to dump everything off, and it’s probably time to grab another archive.

    Most of my music is up on the Internet Archive, which is likely to outlast me, and that’s OK. Should consider packaging more of the tracks on Soundcloud into albums.

    I’ve lost all of my archived data from the mainframe era, and I’m a bit sad about that; there was some really elegant stuff in there — elegant for OS/360 and MVS, I guess…

    I’ve shrunk my physical memorabilia footprint a lot; I have a few things I’d hate to lose, like my board from the 360/95 (did lose my mass store carts and my original FE manual somewhere along the way) and my pocket trumpet, but not as much as I thought before.

    So I think my work for this year will start with finishing up the cleanup of both of our LastPass vaults — that’s mostly done at this point, but making sure we both have a clean copy is a chore — and then finding a way to compile and then deduplicate all those photo libraries (and separate my photos from Shymala’s — we did and still do tend to take shots with each other’s equipment and then forget to split them up).

    I anticipate that job will take quite some time.

    Once that’s done, I’ll come back to the various places my music is stored and get everything out on a release on Bandcamp and the Archive, which will make it available and as safe as I can make it.

    I’m backing up my personal laptop with BackBlaze, which is probably safety enough for most of my data. Will need to review though and make sure it’s all getting backed up. Possibly spending a little to save the various backup disks in BackBlaze is a good idea as well…

    I’ll revisit this over the year, but writing about it helps clarify my thinking some. Back to the passwords.

  • Too long since I contributed to Perl

    I’ve put in two documentation PR’s; funnily enough, I’ve changed email addresses, so now the infrastructure has forgotten that I wrote all the internal comments in the debugger, and I have to wait for someone to trigger the acceptance process.

    Should have done them earlier in the month…

  • Roblox: not a Belloq fail, maybe a me fail?

    In the category of “we can’t handle email right” again, or at least, they haven’t convinced me they can: the email that is this blog’s domain name plus .me.com is apparently on someone’s list of “valid emails you can put in forms”, or there’s a tool that exists somewhere to grab an email off one of the numerous breaches that included it, because it gets used by random people around the world to sign up for stuff.

    This is definitely an “I’m doing this on purpose” because the name is unusual for anyone who doesn’t speak Bahasa Indonesia, and I have never yet had a fraudulent sign-up from Indonesia.

    As I do for my other email, I usually punish them by resetting the password and locking them out of the account. For dating apps I add a really savage profile about how dumb they are.

    But every once in a while there’s one I can’t do this for — Capital One, for instance, allowed ROBIN JEAN (yep, it was all caps) to supply the me.com address as their email for a credit card without verifying that it was accessible by their customer. Their password reset requires, if I recall, the account number to do a reset, so there’s nothing I can do about that one except complain every month when the balance email shows up. (We’re three months in; hasn’t helped, though they keep swearing they’ll fix it.)

    The one I’m writing about today, however,  is one that leaves me gobsmacked. And somewhat alarmed.

    [Edit: I got fooled by a phishing email. Bets that the phone number is bogus and leads to a “why sure we’ll help you, we just need you to click these links…”, but because I am not quite  dumb enough to call random numbers and believe they’re real, I did not get put on a “hey, we got a live one” list, but instead wasted time talking to a robot.]

    On July 1, I got a purchase confirmation from Roblox that read like this (please note that I do not have a Roblox account):

    Thank you for your purchase on Roblox, the online gaming platform that is powering imagination globally!

Please contact us at roblox.com/support, or call us at +1-855-333-4734 if you have any questions about this charge.

Your 6/28/2020 3:11:10 AM order:
Item Purchased: Roblox Premium 2200
Item Price: CAD25.99
Next Renewal Date: 7/28/2020
Total: CAD25.99

Billing Information:
sdf sdf
pemungkah@me.com
Visa ending in 1563
sd
sd
fsd v6e
United States
Username: 45dfgerdfwerewr
Sale ID: 543250908

You will be charged CAD25.99 per month for this service until you cancel. You can cancel at any time by going to the billing tab of the account settings page and clicking cancel membership. If you cancel, you still will be charged for the current billing period. We hope you enjoy your membership!

    Let’s just luxuriate in the utterly transparent fakery of that address and username for a minute. [Edit: which I was supposed to do and panic, not get pissed off and write a lot of emails.]

    It is blatantly obvious that whoever is using this credit card is not on the up-and-up. So I immediately tried to reset the password. Nope. No password reset email. [Edit: this should have been my clue that no such account existed, this was just to reel me in to the bogus “support”.]

    Well, they allow several other authentication schemes, maybe I can’t reset it this way . I’ll make sure that Roblox Support knows about this; possibly unauthorized, fraudulent charges are most certainly going to be a serious issue for Roblox, and they’ll want to be sure that they’ve protected whoever this actually was, and they’ll take quick action to fix this.

    Ha. No. [Edit: Quite embarrassed. I was definitely wrong.]

    I spent the next eleven days simply trying to communicate that someone was very possibly committing fraud, that I had evidence, and that maybe they should do something. {Edit: I did not have evidence. I had a phishing email. I suppose Roblox support shoulda oughta said, “Are you sure this isn’t a phishing email? We don’t have a support phone number/that isn’t our number.”]

    Roblox “support” spent that time sending me their form emails about unauthorized charges. Once I battered my way past that, I said, fine, you can’t tell me anything. Please make sure my email is removed from your system.

    They couldn’t find it.

    I supplied the email with full headers.

    Still couldn’t find it.

    Do you have any explanation as to how this order ended up in my mailbox, then? Because it certainly was not me or anyone in my household. I would think this would be an issue, that there are orders going out to emails that you don’t have any record of.

    Time passes. Crickets.

    Then I get the automated “you haven’t replied and we want to close this ticket so our KPIs look good” email. All right, I will explain it carefully so we can perhaps get an understanding going here.

    Hi. Look. This should not be as hard to understand as it seems to be.

    I forwarded you an email I got. 

    It came to my email address, and had my email address in the purchase record.

    The data in the purchase record is obviously random typing on the keyboard.

    It’s not my credit card.

    It is, however, my email.

    SOMETHING must have created this purchase. 

    [Edit: no, nothing created this purchase, it does not exist. I was barking VERY LOUDLY up the wrong tree.]

    There has to be an audit trail that points back to some account that this purchase order is associated with, and some transaction that initiated it.

    Whatever account it is. Whatever purchase it was.

    NONE OF IT should be associated with my email.

    Have I made it clear?

    Reply:

    To assist with or provide information about any account, we must first verify account ownership. Unfortunately, there is no email address or purchase information associated with the account. Without this information, we are unable to verify ownership or assist further with the account.

Please make sure that with any account you create, you add and verify your email address. This will allow us to verify ownership and also allow you to use the reset password feature.

    What did I just send you, other than the complete email, with all the headers, containing the account name, the email address, the literal transaction ID for the possibly fraudulent sale…? So I gave up.

    [Edit: that should have been the clue that got through: they are saying, there is no such account, we swear; please stop writing.]

    I’m guessing that they may actually have caught that it was bogus right away, and immediately deleted the account, and the stonewalling is to prevent me trying to social-engineer my way into, I don’t know, getting them to confirm the credit card is good or something.

    [Edit: no, Joe, you were really quite wrong and you were just too mad to see it.]

    I’m guessing that there is a  record that this account was deleted because of fraud, but because of policy they can’t tell me that.

    [Edit: THERE IS NO ACCOUNT. IT’S ALL A LIE.]

    But we’ll never know. To whoever owns the credit card, sorry, I did my best. I hope they did protect you, or that you catch the charge and dispute it.

    I’ll just say that I don’t feel warm and fuzzy about the whole thing.

    2025 update: I now think that this might also, possibly, have been a phishing attempt, very badly executed, similar to the 9000 “YOU BOUGHT CRYPTO” / “YOU BOUGHT MCAFEE” scams that go straight to my spam on Gmail. This was on me.com, which has much poorer spam detection.

    If this was a phishing attempt, they should have only supplied the callback number! Contacting support immediately got me “this doesn’t exist”, and in hindsight, probably didn’t!  I was supposed to panic and call the number so they could “help me with my account” (i.e., probably tut concernedly and tell me I “had a virus” and would “need to talk to Microsoft” (I don’t have markup that lets that last bit drip with sarcasm enough.)).

    2026 update: Added all the “I SCREWED UP” edits to make it clear that I was wrong, Roblox was fine, I was having a cranky day/week.

  • An open letter to my namesakes

    Hi, all you other Joe McMahons out there.

    I’m sure you’re all great guys, a pleasure to know, upstanding in your community, and a blessing to your families. Kidding – I know at least one of you wanted to cheat on his wife. Or on someone else’s spouse with their wife.

    “Why would you say a thing like that! How could you possibly know that?”

    Well, Google tells me. I take that back – actually, you tell Google to tell me, via Gmail.

    Many of you seem to have the bad habit of using our name @ gmail.com when you are asked for an email address. I am at a loss as to why you do; I find it hard to believe that you really think that magical email fairies are going to deliver you mail…somehow… because you type in “my.name@gmail.com”. I’m guessing it’s just plain laziness and not being willing to bother to spend the grueling 2 minutes it takes to set up a Gmail account.

    This is why all the places you set up accounts for always break, and you can’t log in anymore. Because it’s my frigging email account, you dolt.

    Whenever you set up an account to post a personal ad in Liverpool, or sell your used van, or buy a new laptop and set up a Windows Live account, or join Ashley Madison (that was my very favorite) and use that account, I’m the one who gets the confirmation message.

    You know what I do? I confirm your account, and then I immediately reset the password on your account so it’s my account, and I set up 50-character passwords and 2-factor authentication if possible.

    You know why? Because accounts linked to my Gmail account might be usable to social-engineer someone into access to my Gmail account.

    Shorter version? ‘Cause fuck you, that’s why.

    And that’s my Gmail account. Keep your grubby paws off it. If not, well, in the words or René Belloq:

    “Again we see there is nothing you can possess which I cannot take away.”

  • Sleep Number beds, QA techniques, and (not) saving $300

    We’ve owned a Sleep Number bed since the mid-1990’s – an original two-cell queen, with the original old model wired-remote, loud-as-hell pump. It’s served us well and is still functioning fine after many years of service. When Shymala moved to Connecticut in 2010, she bought a new single-cell full-sized mattress, and took it with her when she moved to Brooklyn in 2012.

    This last winter, we moved her Sleep Number bed from Brooklyn to San Jose. In Brooklyn, it had worked fine: it stayed inflated well and was solid and comfortable. We carefully did our research about moving inflatable beds and noted that the consensus was that the bed should be deflated to about 30 (out of 100) while going over the Rockies; the change in altitude would cause the air inside at sea-level pressures to expand significantly, and since the bed was quite new, we wanted to make sure it was going to survive the trip.

    We followed the suggested procedure: deflated the bed, check; removed the hose, check; capped it off with the supplied plastic cap, check. Very much like the previous move of this same bed from Connecticut to Brooklyn, which worked just fine. We bagged it up and the movers put it in a box, and off it went to California.

    We set it up on the bed frame in the same way it had been used in Brooklyn, reconnected the pump, and inflated it. Halfway through the first night, Shymala found herself all the way down on the slats of the bed frame. The cell had almost completely deflated. We tried it again, and it didn’t deflate as badly, but still too much to be comfortable. We experimented with how much it was inflated, whether rolling around on the bed made it deflate, but couldn’t get any kind of consistent result.

    We finally hit on a temporary solution of inflating it to somewhat above the desired firmness, then popping the hose off and installing the cap. This kept the cell reasonably well inflated, but it still deflated slowly over the course of a couple days.

    At this point we suspected that there had to be a slow leak in the cell but we couldn’t figure out why. We called Select Comfort, and they were as mystified as we were, which wasn’t encouraging. Their suggestion was that we needed a new cell, so we ordered one. After getting off the phone, we started thinking that since the cell was holding pressure better with the cap on than with the pump, maybe it was the pump instead. So we called back and ordered a pump.

    The pump came, and we tested it with the pump. Same result: inflating it to full then waiting resulted in the cell slowly deflating over an hour or so to a setting of 5 on the remote. This meant the pump was not the problem.

    The pump’s been used for about an hour, so we thought, “great, we can return the pump since it’s not the problem.” We called Select Comfort to arrange a return. Unfortunately, the customer service reps did not mention when we purchased the pump and cell that any parts you buy are non-returnable.

    Let me emphasize that, for anyone else trying to debug a Select Comfort problem: items are 100% non-returnable. Do not expect to be able to swap parts to diagnose a problem.

    So there’s $118 gone west, and the bed is still screwed up. They did offer to let us return the cell since the reps hadn’t told us that about the “you bought it, you’re stuck with it” policy. We hung up, and then started trying to figure out how to debug the problem without blowing any more money.

    At this point, we could not be sure where the problem lay for certain. Was it that the pumps both had a similar issue that our configuration was exposing, or was the cell really messed up somehow in a non-obvious way? This resembles the case of a client-server problem in software: the two aren’t communicating, and you don’t have a way to unit test the client and server.

    We had unit-tested the cell as best we could: we inflated it, and used the cap to simulate the pump sitting there not running. The slow deflation didn’t tell us anything because we weren’t sure that the two situations were exactly equivalent. We didn’t have a way to unit-test the pump (such as a manometer to verify that the check valve in the pump wasn’t leaking). This meant we had to come up with a way to do a better integration test.

    After some thinking, we came up with the idea that there was a second Sleep Number bed available with a different model of pump altogether. If we swapped the pumps, we’d know whether the pump had an issue or if the cell did. (If the new pump held pressure on the old bed, the pump was OK; if the old pump failed to hold pressure on the new cell, the cell was bad.)

    The test showed that the cell was bad: the new pump held pressure on the old bed just fine, and the old pump slowly lost pressure on the new bed.

    Okay, so that meant that if we had opened the cell first, we would have not needed to open the pump at all. Bad guess on our part! We opened the new cell and installed it on the bed. The hose was considerably more difficult to get on the new cell, and observation showed that there were two O-rings that…aw, crap. I finished installing the hose, pumped up the bed, and as expected, it held pressure just fine with the original pump that we’d brought from Brooklyn.

    I then went out and looked at the old cell. The attachment had a small groove in it which contained…no O-ring. Apparently when we disassembled the bed in Brooklyn, the O-ring failed or was pulled off. The connection in California had a good-enough seal to inflate the bed and keep it inflated for a while, but not a good enough one to sleep on. We essentially ended up spending $200 to replace a 25-cent O-ring.

    Conclusions:

    • Make sure you know baseline conditions before you start testing – the classic “known-good state”. If we’d had a baseline state that included a check for the O-ring, we would have solved this by a short trip to the hardware store.
    • Make sure that your oracle, if you have one, is sufficient. Select Comfort’s customer service does not have a good diagnostic tree available to help them spot this problem, and therefore couldn’t tell us to check for this particular issue.
    • Make sure you know the costs for testing. In this case,  swapping parts to fix a Select Comfort bed is a problem, as everything is non-returnable.
    • Know your problem space. When looking at a pneumatic connection, check to see if there are supposed to be O-rings there and verify that they’re still there.
    • Search for the problem; if you have to solve it yourself, document it. No search turned up “look for the O-ring”, so here’s a tip: if you have an intermittent deflation problem with a Select Comfort bed after a move, look for the missing O-ring first.