Before the Shadow Play

Author: Joe McMahon

  • obliquebot returns

    Some time back, when beepboop.com was still around, I wrote a little Slack bot that listened for “oblique” or “strategy” in the channels it had been invited to, and popped out one of Eno’s Oblique Strategies when it heard its keywords or was addressed directly.

    It worked fine up until the day that BeepBoop announced that they were going away, and eventually obliquebot stopped working.

    This month, I decided that I would stop ignoring the “you have a security issue in your code” notifications from GitHub, and try catching obliquebot up with the new version of the SLAPP library that I’d used to get Spud, the RadioSpiral.net “who’s on and what’s playing” robot back online.

    I went through all the package upgrades and then copied the code from Spud over to the obliquebot checkout. The code was substantially the same; both are bots that listen to channels and respond, without doing any complex interaction. I needed to add the code to load the strategies from a YAML file and to select and print one, but the code was mostly the same.

    I also needed to update the authentication page to show the obliquebot icon instead of the RadioSpiral one, and to set the OAuth callback link to the one supplied by Slack.

    Once I had all that in place, I spent a good two or three hours trying to figure out why I could build the code on Heroku, but not get it to run. I finally figured out that I had physically turned off the dyno, and that it wasn’t going to do anything until I tuned it back on again.

    obliquebot is now running again at RadioSpiral and the Disquiet Junto Slack, and I’ve updated the README at the code’s GitHub page to outline all the steps one needs to take it and build one’s own simple request-response bot.

  • Show report: 2020-10-31 “Pharoah Nuff” at radiospiral.net

    My last performance was not as smooth as I hoped, so this time I decided that I would find a way to streamline it even further.

    I decided to go further in the direction I’d taken with the Wizard of Hz show, and strip down even more. I decided to try to perform as much as possible of the set on the iPad, and use the laptop solely for streaming and Second Life. This freed me from hassles in switching setups in VCVRack, Live, and the other software I’d been using, but it also meant that I wouldn’t be using either of my favorite synths for this performance (the Arturia 2600 and Music Easel).

    Having had some time between performances to really experiment with AUM and I felt comfortable using it to lay out my performance. I decided that I wanted to keep Scape as my background/comping program, and that I’d set up a series of light-handed scapes to give me a through-line. I then sat down with MIRack and Ripplemaker to create multiple Krell textures that I could bring in and out, and also discovered a couple of lovely lead patches for Ripplemaker that I paired with a Kosmonaut looper. I also brought in a couple public-domain samples from old sci-fi movies, heavily processed with Kosmonaut again, and felt like I had enough material to do an hour’s performance.

    I used the iConnect Audio4+, which I now finally have the hang of, and set it up so that I had two stereo channels from the iPad and one mono channel routed to the iPad through Kosmonaut (again!) for some subtle reverb when I was doing my intro and outro. With the setup I used, the iConnect kept the iPad fully charged through the whole set.

    I used Loopback to connect the multiple outs from the iConnect to the stereo ins on my Mac, and monitored on headphones. I pulled up Audio Hijack, entered the stream setup, and was ready to broadcast.

    I got up early on the day, started up AUM, and ran a soundcheck to make sure everything was working. All sounded good, and I was good to go.

    Mostly.

    I didn’t stop AUM, and as a result, it ran for several hours before I tried to start using it. This apparently triggered some kind of a memory shortage, and when I started streaming, I was completely mute. Fortunately, I’d cued up a prerecorded VCVRack texture, and started that while I was trying to figure out what was wrong. I gave up and restarted the iPad, and AUM came up like a champ.

    After that it was pretty smooth. I was able to fade the various patches in and out, play the sci-fi samples, and improvise over the Scape-provided background. Once it was off the ground, the performance was very easy to do. I did forget and leave the audio feed from Second Life enabled, so as a result this was a very sparse performance, but the sparseness worked out very well.

    Overall this was a great way to do a performance and I plan to refine this further. Of particular note is that AUM saves things so well that it will be trivially easy to do this performance again, should I decide to; this is probably the first time I’ve had a performance setup I felt was robust enough to say that!

  • RadioSpiral Wizard of Hz Performance Notes

    Last time I did a live streaming performance for an audience, it did not go well. I had long pauses, the mic didn’t work, and miscommunication over Slack to the remote venue resulted in my getting cut off before my set was finished. And this was even after a good bit of practice.

    So when I signed up for the Wizard of Hz concert on RadioSpiral, decided that I needed to have as much backstop as possible in place so that no matter how tangled up I got mentally, I’d have a fallback to something that sounded good and would be a nice navigable arc from point A to point B. Ideally, I should have something that would sound great even if I got called away for the entire set!

    My go-to process for this is Scape. I’ve had it since it first came out, and it meshes very well with what I enjoy hearing and enjoy playing. I started off with the Scape playlist that I often use to relax and get to sleep; this is a seven-scene playlist, with the transition time at max, with the per-scene time adjusted to be just a bit over an hour. This gives me a fallback for the whole hour; I can pull everything else back and lean on Scape while I decide what the next section should be.

    In addition, Scape provides a very nice backdrop to improvise over, so I can be playing something while Scape gives me a framework.

    I then put together a couple of Ableton Live sets: one built on the Arturia ARP 2600 and Buchla Music Easel emulations, and another built on Live’s really nice grand piano and the open-source OB-Xa emulator, the OB-Xd. I finally figured out how to change patches on the OB-Xd about 20 minutes before showtime.

    I had set up a piano with a nice looping effect from Valhalla Supermassive (Supermassive and Eventide Blackhole figured heavily in the effects), but ended up not using it, and doing a small Launchpad set instead using the Neon Lights soundpack.

    I was also able to open and close with the large singing bowl, played live and processed through the Vortex, which was a nice real analog performance touch.

    Overall, I strove for a set that sounded played-through, but that had enough breathing room that I could fall back on Scape while making changes (switching Live sets, etc.), and I think I achieved that.

    I did have Audio Hijack recording the set, so if it sounds OK, I’ll be releasing it on Bandcamp. (Followup: it came out pretty well! Definitely at least an EP.)

    Only real issue was a partially-shorted cable between my iPhone and the mixer that I didn’t figure out until most of the way through the set.

  • Squaring numbers and a forgotten book

    I happened on a demonstration of a mental math trick on Reddit for squaring numbers in your head and was immediately reminded of a technique I learned in 1972 from a great book on speed arithmetic that I have unfortunately forgotten the name of.

    The video’s formulation uses the identity n^2 = (n^2 - a^2) + a^2 to make the multiplication simpler, but the book had an extremely elegant way to notate a different identity that works nicely for doing the squares of two-digit numbers in one’s head, and rapidly doing multi-digit squares on paper.

    The Reddit example squared 32 by changing it to 32 * 32 = 30 * 34 + 4 = 1024, which is clever, but check this out!

    Start with the identity (a + b)^2= a^2 + 2ab + b^2 and treat 32 * 32 as (30 + 2)^2.

    Visualize this in your head:

    0904 
 12

    That’s the a^2 + b^2 on the first line, and 2ab on the second. Now just add it up normally, with blank spaces equal to zeroes, and you get 0, 10, then 102, then 1024.

    The left-to-right add means you never have to remember the carry value, just the changed result. Let’s try 47.

    1649
 56

    1, 21, 220, 2209. Simple.

    The Wikipedia page on mental arithmetic is a great resource that has this technique, but lacks the notation visualization shown here which honestly is what makes it easy. The same technique works for larger numbers too. There’s more to remember, which may make it too hard to do in your head, but it makes squaring large numbers on paper trivial.

    Let’s say we want to square 123:

    010409 
 0412 
  06

    1, 14, 151, 1512, 15129. (a^2 + b^2 + c^2 + 2ab + 2bc + 2ac). Squares on the top row, 2ab on the left in the middle, 2bc on the right in the middle, 2ac on the bottom.

    I will admit that I didn’t properly get how to do the multi-digit notation right 45 years ago, but I hadn’t really understood the mapping of the identity to the positions on the page and was doing it by rote. The notation is the slickest part of this, as it automatically handles the proper number of multiplications by 10 for you.

    The left-to-right addition and a trick of doing mental addition by repeating the current total to oneself when adding the next number to keep from losing one’s place (ex. 45 + 37 + 62 – 45, 75, 75…82, 142, 144 and cast out 9s — 0, 1, 9, and 1+4+4 =9) were all also in that same book. I really wish I could remember what it was!

  • Belloq fail: Roblox

    In the category of “we can’t handle email right” again, or at least, they haven’t convinced me they can: the email that is this blog’s domain name plus .me.com is apparently on someone’s list of “valid emails you can put in forms”, or there’s a tool that exists somewhere to grab an email off one of the numerous breaches that included it, because it gets used by random people around the world to sign up for stuff.

    This is definitely an “I’m doing this on purpose” because the name is unusual for anyone who doesn’t speak Bahasa Indonesia, and I have never yet had a fraudulent sign-up from Indonesia.

    As I do for my other email, I usually punish them by resetting the password and locking them out of the account. For dating apps I add a really savage profile about how dumb they are.

    But every once in a while there’s one I can’t do this for — Capital One, for instance, allowed ROBIN JEAN (yep, it was all caps) to supply the me.com address as their email for a credit card without verifying that it was accessible by their customer. Their password reset requires, if I recall, the account number to do a reset, so there’s nothing I can do about that one except complain every month when the balance email shows up. (We’re three months in; hasn’t helped, though they keep swearing they’ll fix it.)

    The one I’m writing about today, however,  is one that leaves me gobsmacked. And somewhat alarmed.

    On July 1, I got a purchase confirmation from Roblox that read like this (please note that I do not have a Roblox account):

    Thank you for your purchase on Roblox, the online gaming platform that is powering imagination globally!

Please contact us at roblox.com/support, or call us at +1-855-333-4734 if you have any questions about this charge.

Your 6/28/2020 3:11:10 AM order:
Item Purchased: Roblox Premium 2200
Item Price: CAD25.99
Next Renewal Date: 7/28/2020
Total: CAD25.99

Billing Information:
sdf sdf
pemungkah@me.com
Visa ending in 1563
sd
sd
fsd v6e
United States
Username: 45dfgerdfwerewr
Sale ID: 543250908

You will be charged CAD25.99 per month for this service until you cancel. You can cancel at any time by going to the billing tab of the account settings page and clicking cancel membership. If you cancel, you still will be charged for the current billing period. We hope you enjoy your membership!

    Let’s just luxuriate in the utterly transparent fakery of that address and username for a minute.

    It is blatantly obvious that whoever is using this credit card is not on the up-and-up. So I immediately tried to reset the password. Nope. No password reset email. Well, they allow several other authentication schemes, maybe I can’t reset it this way . I’ll make sure that Roblox Support knows about this; possibly unauthorized, fraudulent charges are most certainly going to be a serious issue for Roblox, and they’ll want to be sure that they’ve protected whoever this actually was, and they’ll take quick action to fix this.

    Ha. No.

    I spent the next eleven days simply trying to communicate that someone was very possibly committing fraud, that I had evidence, and that maybe they should do something.

    Roblox “support” spent that time sending me their form emails about unauthorized charges. Once I battered my way past that, I said, fine, you can’t tell me anything. Please make sure my email is removed from your system.

    They couldn’t find it.

    I supplied the email with full headers.

    Still couldn’t find it.

    Do you have any explanation as to how this order ended up in my mailbox, then? Because it certainly was not me or anyone in my household. I would think this would be an issue, that there are orders going out to emails that you don’t have any record of.

    Time passes. Crickets.

    Then I get the automated “you haven’t replied and we want to close this ticket so our KPIs look good” email. All right, I will explain it carefully so we can perhaps get an understanding going here.

    Hi. Look. This should not be as hard to understand as it seems to be.

I forwarded you an email I got. 

It came to my email address, and had my email address in the purchase record.

The data in the purchase record is obviously random typing on the keyboard.

It’s not my credit card.

It is, however, my email.

SOMETHING must have created this purchase. There has to be an audit trail that points back to some account that this purchase order is associated with, and some transaction that initiated it.

Whatever account it is. Whatever purchase it was.

NONE OF IT should be associated with my email.

Have I made it clear?

    Reply:

    To assist with or provide information about any account, we must first verify account ownership. Unfortunately, there is no email address or purchase information associated with the account. Without this information, we are unable to verify ownership or assist further with the account.

Please make sure that with any account you create, you add and verify your email address. This will allow us to verify ownership and also allow you to use the reset password feature.

    What did I just send you, other than the complete email, with all the headers, containing the account name, the email address, the literal transaction ID for the possibly fraudulent sale…? So I gave up.

    I’m guessing that they may actually have caught that it was bogus right away, and immediately deleted the account, and the stonewalling is to prevent me trying to social-engineer my way into, I don’t know, getting them to confirm the credit card is good or something.

    I’m guessing that there is a  record that this account was deleted because of fraud, but because of policy they can’t tell me that.

    But we’ll never know. To whoever owns the credit card, sorry, I did my best. I hope they did protect you, or that you catch the charge and dispute it.

    I’ll just say that I don’t feel warm and fuzzy about the whole thing.

  • The Harp of New Albion’s Tuning for Logic

    The Disquiet Junto is doing an alternate tunings prompt for week 0440 (very apropos!).

    I’ve done several pieces before using Balinese slendro and pelog tuning, most notably Pemungkah, for which this site is named. I wanted to do something different this time, using Terry Riley’s tuning from The Harp of New Albion, using Logic Pro’s project tuning option.

    The original version was a retuning of a Bosedorfer grand to a modified 5-limit tuning:

    However, Logic’s tuning feature needs two things to use a tuning with it:

    • Logic’s  tuning needs to be based on C, not C#
    • The tuning has to be expressed as cents of detuning from the equal-tempered equivalent note.

    This leads one to have to do quite a number of calculations to put this in a format that Logic will like.

    (more…)

  • Life in the fast lane / Surely makes you lose your mind

    I came back to the Radiospiral iOS app after some time away (we’re trying to dope out what’s going on with metadata from various broadcast setups appearing in the wrong positions on the “now playing” screen, and we need a new beta with the test streams enabled to try things), only to discover that Fastlane had gotten broken in a very unintuituve manner. Whenever I tried to use it, it took a crack at building things, then told me I needed to update the snapshotting Swift file.

    Okay, so I do that, and the error persists. Tried a half-dozen suggestions from Stack Overflow. Error persists. I realized I was going to need to do some major surgery and eliminate all the variables if I was going to be able to make this work.

    What finally fixed it was cleaning up multiple Ruby installs and getting down to just one known location, and then using Bundler to manage the Fastlane dependencies. The actual steps were:

    1. removing rvm
    2. removing rbenv
    3. brew install ruby to get one known Ruby install
    4. making the Homebrew Ruby my default ( export PATH=/usr/local/Cellar/ruby/2.7.0/bin:$PATH)
    5. rm -rf fastlane to clear out any assumptions
    6. rm Gemfile* to clean up any assumptions by the current, broken Fastlane
    7. bundle install fastlane (not gem install!) to get a clean one and limit the install to just my project
    8. bundle exec fastlane init to get things set up again

    After all that, fastlane was back to working, albeit only via bundle exec, which in hindsight is actually smarter.

    The actual amount of time spent trying to fix it before giving up and removing every Ruby in existence was ~2 hours, so take my advice and make sure you are absolutely sure which Ruby you are running, and don’t install fastlane into your Ruby install; use bundler. Trying to fix it with things going who knows where…well, there’s always an applicable xkcd.

    You are in a maze of Python installations, all different

  • Broken iframes and HTML::TreeBuilder

    We had a situation last week where someone had entered a broken <iframe> tag in a job description and our cleanup code didn’t properly remove it. This caused the text after the <iframe> to render as escaped HTML.

    We needed to prefilter the HTML and just remove the <iframe>s. The most difficult part of this was figuring out what HTML::TreeBuilder was emitting and what I needed to do with it to do the cleanup. It was obvious that this would have to be recursive, since HTML is recursive (there could be nested, or multiple uncosed iframes!) and several tries at it failed until I finally dumped out the data structure in the debugger and spotted that HTML::TreeBuilder was adding “implicit” nodes. These essentially help it do bookkeeping, but don’t contain anything that has to be re-examined to properly do the cleanup. Worse, the first node contains all th text for the current level, so recursing on them was leading me off into infinite depths, as I kept looking for iframes in the content of the leftmost node, finding them, and uselessly recursing again on the same HTML.

    The other interesting twist is that once I dropped the implicit nodes with a grep, I still needed to handle the HTML in the non-implicit nodes two different ways: if it had one or more iframe tags, then I needed to use the content method to take the node apart and process the pieces. There might be one or more non-iframes there, which end up getting returned untouched via as_HTML. If there are iframes, the recursion un-nests them and lets us clean up individual subtrees.

    Lastly, any text returned from content comes back as an array of strings, so I needed to check for that case and recurse on all the items in the array to be sure I’ve filtered everything properly. My initial case checks for the trivial “no input so no output”, and “not a reference” to handle the starting string.

    We do end up doing multiple invocations of HTML::TreeBuilder on the text as we recurse, but we don’t recurse at all unless there’s an iframe, and it’s unusual to have more than one.

    Here’s the code:

    +sub _filter_iframe_content {
  my($input) = @_;
  return '' unless $input;

  my $root;
  # We've received a string. Build the tree.
  if (!ref $input) {
    # Build a tree to process recursively.
    $root = HTML::TreeBuilder->new_from_content($input);
    # There are no iframe tags, so we're done with this segment of the HTML.
    return $input unless $root->look_down(_tag=>'iframe');
  } elsif (ref $input eq 'ARRAY') {
    # We got multiple strings from a content call; handle each one in order, and
    # return them, concatenated, to finish them up.
    return join '', map { _filter_iframe_content($_) } @$input;
  } else {
    # The input was a node, so make that the root of the (sub)tree we're processing.
    $root = $input;
  }

  # The 'implicit' nodes contain the wrapping HTML created by
  # TreeBuilder. Discard that.
  my @descendants = grep { ! $_->implicit } $root->descendants;

  # If there is not an iframe below the content of the node, return
  # it as HTML. Else recurse on the content to filter it.
  my @results;
  for my $node (@descendants) {
    # Is there an iframe in here?
    my $tree = HTML::TreeBuilder->new_from_content($node->as_HTML);
    if ($tree->look_down(_tag=>'iframe')) {
      # Yes. Recurse on the node, taking it apart.
      push @results, _filter_iframe_content($node->content);
    } else {
      # No, just return the whole thing as HTML, and we're done with this subtree.
      push @results, $node->as_HTML;
    }
  }
  return join '', @results;
}

  • Fixing a commit in the middle of a set

    Another tip for those who’ve needed to do this: let’s say you’ve created a feature branch and are adding tests to the code, and you realize that one of your tests is incorrect several commits further on. What do you do?

    If this is a plain old feature branch, you can just make the fixup commit and have two commits for that test. This is perfectly fine.

    If however, you’re constructing a series of commits to be cherry-picked later, it’s better to have all the related changes together.

    You can do this by doing a git log and capturing the output in order back to the incomplete commit. Save that output, then git reset --hard oldcommit.

    The incomplete commit is now the current one, so you can make any fixes you need, git add them, and then git commit --amend to include them in the current (formerly incomplete) commit.

    Now go back to the log output, and find the commit just after the current one; record that, and then record the old HEAD commit. git cherry-pick nextcommit..oldhead will then reapply all of the commits following the one you just repaired, and your branch will be back where it was, with the one incorrect commit repaired.

  • Avoiding gigantic changesets when refactoring

    This may seem obvious to many folks but for those who haven’t had to do something like this before, but I think it’ll be very illuminating to those who haven’t.

    The starting point and a naive solution

    I’m currently working on a cleanup in our codebase at ZipRecruiter, where we want to remove dependencies shared between two parts of the codebase into a common area. The part I’m working on now is well-defined, but touches a very large number of modules throughout the codebase.

    If I chose a naive way of doing the port, I’d do the the following:

    1. Port the functions to be moved to a new module in the common area and ensure they’re all under test.
    2. Go through the rest of the codebase, find all references to the ported functions, and update the modules to use the new module instead of the old.
    3. Remove the functions from the old module now that all the other code has been ported.

    If the functions in the old module aren’t used much, then step 2 is fine as is, but this is a utility module that’s used in a lot of modules. This makes step 2 a gigantic risk point because the changeset for a large number of modules essentially has to be done all at once, whether we do it as one big commit or a lot of small ones that still have to be applied at the same time.

    Doing these all at once is going to be a code review and QA nightmare, so this isn’t going to fly.  Unfortunately, I started on the port and didn’t come to this understanding until I’d gotten pretty far into the changes. I needed to save the work I’d done, but reuse it in a way that was safe and not an unbearable load for my reviewers and QA.

    (more…)