Before the Shadow Play

Blog

  • sd1diskutil: Resurrecting a disk format from the dead for a brand-new emulator

    A little history

    The Ensoniq SD-1 is a synthesizer from the 1990s — a ROMpler with a Motorola 68000 at its heart. Like many synthesizers of the era, it uses the cheap, easy, and simple storage medium of the day: 800K floppy disks for storage of everything: factory programs, user programs, presets (organized small sets of programs), full-up MIDI sequences, and its own operating system.

    The format is proprietary and somewhat peculiar: a custom FAT, 10 sectors per track numbered 0 through 9 (not 1 through 10 like a PC), big-endian multi-byte fields throughout, and a handful of file types that the rest of the computing world has never heard of. To retrieve your data on a modern computer, or to get sounds back onto the synth, you need something that can speak this format.

    Few if any USB disk drives can handle this format; the extant programs which can read Ensoniq disks all run under MS-DOS (or Windows DOS emulation) and need a real, wired-in diskette drive to handle reading and writing disks. Forget about doing this on a Mac.

    Fortunately, the SD-1 has a reasonably robust MIDI system-exclusive, or “SysEx”, implementation, capable of dumping and receiving pretty much everything except the actual sequencer OS that can record sequences to the SD-1’s internal memory and play them back. Those of us who saw the handwriting on the wall (and who didn’t want to keep a 486 tower lying around just to write the floppy disks that were becoming harder and harder to find anyway), took the earliest possible opportunity to dump everything out over SysEx and save it elsewhere.

    Getting the sequencer OS back into the thing still needs a diskette, which is an issue (solved by third-party add-ons that could store hundreds of floppy images on a USB stick).

    The renaissance

    But there was some big news in March 2026, that made the question of accessing the SD-1’s disks and data an interesting topic again.

    The folks at Sojus Records announced a wrapper around the previously-created SD-1 MAME emulator that allowed the SD-1 to be loaded as a VST3 plugin.

    For all of us who had SD-1’s (or who still have them, but have shifted to much-more-convenient computer-based sequencing), this was a sit-up-and-take-notice moment. Our baby was now a plugin! And all that work we’d done previously was now usable again.

    However! The first release of the plugin was only able to read .IMG files — a file format created by Gary Giebler to store floppy images on disks other than floppies. This meant that there needed to be a way to get .syx SysEx files back onto .IMG images so they could be used once more.

    Sure, the Giebler and Rubber Chicken utilities were still out there, but I’m a Mac guy, and attempts to get those running properly on emulated MS-DOS were pretty much a failure. What I needed was a utility that could read and write disk images on my Mac.

    A year ago I would have looked at that and said, “man, I do not have the time or the patience to read all those Transoniq Hacker articles and try to piece this together.” This year, I didn’t have to have that patience: I had Claude, and $20 worth of tokens a month to spend, so I thought, why not? This is actually a fairly well-defined problem:

    • Documentation for the disk organization and file formats exist in this PDF archive of the Transoniq Hacker
    • We have some disk images that we know work with the emulator, including a sequencer OS disk
    • The emulator seems to be able to read .IMG files fine, so if I can figure out how to write disks, I should be able to read them on the emulator.

    This is a pretty solidly mapped-out basis to start from, and I figured that with both good documentation, sample data, and a working system to test against, I stood a pretty good chance of being able to carefully steer Claude to a solution.

    Getting started

    I decided that I wasn’t going to be fancy here. This is going to be called sd1diskutil because it’s just going to be a wrapper around a library that knows how to do the job.

    So on March 26th, I sat down with Claude in the terminal, loaded obra/superpowers, and started brainstorming. I decided that, contrary to my more recent utility projects, I’d try to produce something that could be embedded into a prettier interface than just the command line.

    That meant the first decision was to what language to implement this in, and after some discussion with Claude, we settled on Rust, to allow me to make this a functional programming approach, using very tight types and operations on them. This was, in hindsight, probably colored by my experiences with Scala and really tight types, and how that made it so much easier to write correct code.

    To go with that and make it usable, we came up with a very thin CLI binary (sd1cli) that could convert MIDI SysEx dumps to and from the SD-1’s on-disk binary format provide full disk management — list, inspect, write, extract, delete, create.

    Because I knew that eventually I wanted to wrap this up in a pretty UI, I asked Claude how to build Swift bridging in, and it included a clean UniFFI surface for SwiftUI, allowing the same library to eventually power a macOS application.

    I knew that I was going to have to be very careful to implement this correctly. File systems and custom file formats are not forgiving, and the SD-1’s OS, though quite capable, is not what you would call robust.

    • The architecture therefore mirrored the data as closely as possible and tried to ensure that everything was as safe and stable as possible:
    • DiskImage owns the raw 819,200-byte image.
    • FileAllocationTable is a stateless handle that operates on a &mut DiskImage to avoid Rust borrow conflicts.
    • SubDirectory follows the same pattern.
    • SysExPacket is the only place nybble encoding and decoding happens — every layer above it works in plain bytes.
    • Atomic writes everywhere: save to a temp file, then rename.

    I started out with a blank disk template created by the SD-1 emulator. What better source for a good disk than the emulator itself? (Oh, you sweet summer child. We’ll spend about three days beating our heads against this disk image.)

    The first working implementation was achieved in a single burst: we planned out the types and operations, reviewed the design, and then built an implementation plan: workspace scaffold, error types, disk image, FAT, directory, SysEx parser, domain types, the full CLI. Integration tests for every command. The code compiled. The tests passed.

    Then came the reality checks.

    Block 4? Or block 5?

    The Giebler articles said that the FAT should be at block 5. But our empty disk image from the emulator said it was at block 4. Everything else matched up: ten blocks long, 170 three-byte entries each. What was the problem? We could write files with our code to the blank image, and the SD-1 emulator could read them.

    Figuring this took way longer than it should have, for a reason that only became clear in retrospect.

    The blank disk template used during early development, as mentioned above, had been written by the Sojus VST3 plugin, which it turned out was concealing a sector-shifting bug that was actually happening at the underlying MAME emulator level!

    See, normal DOS disks have 11 sectors per track, numbered 1 through 11. The SD-1 disks have ten, numbered zero to 9! MAME does handle the ten sectors per disk thing fine…but it uses SD-1 blocks 1 though 9, dropping block 0 and adding an empty all-zeroes block 10. So a fresh emulator-written disk has its FAT at block 4 instead of block 5. And a freshly-written emulator disk also immediately throws a DISK ERROR – BAD FORMAT if you try to read it…but we were only trying to write it, thereby breaking it, and then read it with our Rust code!

    The Giebler article in Transoniq Hacker said block 5 and our write code was written to put it at block 5, which was correct. The disk from the emulator said block 4, because the emulator dropped block 0. The emulator could read the test disks written by our code fine — because sd1diskutil‘s own writes were correct.

    But whenever the emulator saved a file back, it would quietly apply the shift again, moving the FAT back to block 4, exactly matching the initial (broken, but we didn’t know it) blank disk. So we went round and round, trying to resolve this: the article says 5, the disk says 4, the emulator reads 5 and writes 4, maybe both are correct and the article didn’t have that, so we should support either, or…?

    I repeatedly tried to add files to the disks written by the emulator, and they always got DISK ERROR – INVALID FORMAT. How was I screwing this up?

    I finally figured it out when I wrote a file to a good (block-5) disk and immediately tried to read it back (from the now block-4 disk). The emulator immediately threw a BAD DISK error…on its own output! So the emulator was wrong (though at the time we didn’t know why — see below!), and the article was right.

    We created an empty disk by taking a copy of the known-good SEQUENCER-OS disk and deleting all the files from it using our code. We then wrote a single file to it and tried it on the emulator…and the disk was readable and the file was there.

    Other early problems and fixes followed the same discipline of checking what we wrote against the emulator: local filenames needed to be forced to uppercase because the SD-1’s LCD doesn’t render lowercase. We had to analyze AllPrograms and AllPresets files on the SD-1 SEQUENCER-OS disk to figure out how to write them, and analyze how these were encoded into SysEx types (the SD-1 MIDI implementation helped some, but a lot had to be worked out from just trying things until we worked them out). The free block count management logic needed a rewrite. Each fix came from trying it and checking it against what the emulator would accept, not just blindly accepting the documentation, useful though it was,

    The Program Interleave Bug

    Of all the bugs, the program interleaving bug was the most insidious and hardest to fix, because it was wrong in a way that “worked”. Nothing broke, the OS didn’t complain, and on first glance seemed to be completely reasonable and perfectly correct…but sequences played back with all the wrong programs if they were being loaded from program memory. (ROM patches were fine.)

    Programs stored in a SixtyPrograms file are byte-interleaved on disk: there are two independent 15,900-byte streams packed together, with the even byte positions carrying programs 0 through 29, and the odd byte positions carrying programs 30 through 59.

    The original SysEx-to-program bank implementation had picked then outas alternating pairs — taking a program from the first half for bank 1, program 1, then a program from the second half for bank 1, program 2, and so on. The result was that programs were extracted perfectly, but landed at the wrong bank and patch position.

    I was able to figure this out by loading a custom 60-sequence file with 60 embedded programs, playing it back, and seeing that the expected patches were there but in the wrong places. Unfortunately, the fact that the last time I’d actually looked at these sequences was 2010 or so meant that I didn’t remember where the right position was! I knew they were wrong because they sounded wrong, but not where the right place was.

    It wasn’t until I found a SysEx dump of one of the factory sample sequences that we were able to write that to a .IMG and load it, then compare the locations where the programs went when our file was loaded in contrast to where they ended up when they were loaded from the SEQUENCER-OS disk by writing them down by bank and slot both ways and letting Claude figure out the mapping, which it did quite nicely.

    Extracting the even and odd byte streams in both the file we wrote and the “good” on-disk one, and then searching for known program names within each stream, allowed Claude to find where the one patch I definitely knew the sequence used was in both sets of interleaved data and then derive the correct mapping: a first-half/second-half split rather than alternating pairs.

    In the process of figuring all this out, we created a Python analysis tool, dump_programs.py, which could extract and list individual programs from multi-program SysEx dumps and disk files. Once we verified the extraction algorithm in the Python code, we were easily able to replicate it in Rust, and test it with two other sequence-and-program dumps by verifying they played back correctly on the emulator after being written to disk from a SysEx file.

    Sequences and a Deeper Problem

    Extracting sequences revealed a gap in the implementation: we could write SysEx files containing sequences and patches, but we couldn’t read them. There was a function that converted SysEx to the on-disk representation, but nothing that went the other direction.

    We discovered this when we realized that extraction was wrapping raw disk bytes in a SingleSequence SysEx header and producing output twice the expected size. Proper test-first setup allowed us to easily implement this properly, writing a 60-sequence-and-program file to disk, verifying it sounded correct on the emulator, and then reading it back out to .syx, verifying that the new file was a byte-for-byte match against the original.

    At this point, I wrote up the block-4/block-5 bug for the Sojus folks to take a look at on GitHub. (At this point I knew that there WAS a bug, but not WHY there was a bug.) They got back to me very quickly, and confirmed that yep, the emulator was screwing up the disk, and why.

    The plugin routes all floppy writes through MAME’s get_track_data_mfm_pc, a function that expects PC-standard sector numbering (1–10). As mentioned, the Ensoniq format uses 0–9. MAME silently discards sector 0 of every Ensoniq track, shifts the remaining sectors down by one, and zeros the last slot! Once the emulator rewrites the track, every block on the track contains wrong data, and every tenth block is zeroed. This was the same bug that had broken the early blank disk template and sent us chasing the wobbly FAT location for days — now fully understood, and confirmed with the Sojus developers, who identified the affected code as esq16_dsk.cpp in MAME — the DOS-to-Ensoniq-and-back block mapping.

    They’re busy working that as of 3/28, but in the meantime, they found a workaround: the emulation code can also read HFE format files. HFE stores the raw MFM flux data and bypasses MAME’s sector enumeration and extraction entirely. Which is totally awesome…but the sd1diskutil code did not speak HFE, and I’d never even heard of HFE. Watching the wonderful Usagi Electric suss out data encoding has educated me a little bit on data transitions and stuff like that, but it wasn’t something I was ready to work on myself at all!

    Archaeology Before Engineering

    Fortunately, the Sojus folks had an HFE image with some data on it to test with: two single patches (OMNIVERSE, SOPRANO-SAX) and a 60-program file. Claude and I embarked on trying to make sense of the data in this file, and this is where Claude seriously impressed me.

    We started off with this HFE file. We knew basically that it should have MFM data in it, and nothing else. Claude bootstrapped up from knowing what MFM data should look like to actually finding it in the file and making sense of this otherwise opaque stream of bits!

    Claude’s first attempt at locating sector headers found nothing. The standard MFM A1* sync marker should have been there ([0x44, 0x89]) but did not appear anywhere in the file. Claude figured out that this was because HFE stores bits LSB-first per byte, in the order the read head encounters them. The standard representation is MSB-first, so at first glance the data made no sense. Claude tried a bit-reversed version of the data, then a bit-reversed-per-byte version, and found the sync marker! [0x22, 0x91], with each byte bit-reversed.

    Once that hurdle was crossed, it was simple for Claude to find the markers and decode all 1600 sectors. The FAT free count in the decoded image matched the hardware OS block count: 1510. The block-to-sector mapping was confirmed against blank_image.img:

    block = track × 20 + side × 10 + sector

    Track geometry was pinned down: each side of a track is exactly 12,522 encoded bytes. The fixed preamble (Gap4a, sync, Gap1) consumes 284 bytes. Each of the 10 sectors is 1,148 bytes with a fixed structure. The remaining 758 bytes are inter-sector gaps — 75 bytes for sectors 0 through 8, and the rest absorbed by sector 9.

    I would have taken quite a long time and a lot of poring over the MFM spec and a lot of trial and error to figure this out, if ever, and Claude had it all doped out in half an hour or so.

    At this point Claude knew how to read an HFE file but not what we should do with it.

    We invoked the brainstorm/design spec/implemetation plan path again. I proposed that what we needed was a translation layer: just get the HFE image to an IMG image, and all of our tools could easily handle it. To use it on the emulator, we’d just convert the IMG image back to HFE, which the emulator should safely be able to read and write.

    Superpowers wrote a complete design spec before any Rust code was touched, pulling in all the information Claude already had at hand about how the HFE files work, writing Python code to cross-check assumptions made in the spec: exact constants, the complete MFM encoding rules, CRC16-CCITT coverage, the interleaved side storage layout, error variants, test cases, the works.

    Then superpowers wrote the implementation plan, with concrete function signatures, and specific expected outputs, all properly built as functions on types and easily testable.

    HFE Implementation, going full vibe

    At this point it was Claude’s party. I had read the spec and the plan, and everything looked reasonable, but I didn’t really know the HFE spec solidly enough to critique the code.

    Claude created three new error variants: InvalidHfe, HfeCrcMismatch, HfeMissingSector, each carrying track, side, and sector context so errors are never ambiguous. Then hfe.rs itself: 771 lines creating the full encode/decode pipeline, and new CLI subcommands hfe-to-img and img-to-hfe to encode and decode HFE images.

    The superpowers code review caught one bug: header offset 17 — the “do not use” field in the HFE v1 spec — was being written as 0x00. The spec requires 0xFF. A strict HFE reader would reject the file, and we knew the right answer, so…easy fix.

    After implementation, the code passed all the low-level tests, and read_hfe on the sample HFE file properly decoded all 1600 sectors, returning a DiskImage whose directory listed OMNIVERSE, SOPRANO-SAX, and 60-PRG-FILE with the correct free block count. A complete round-trip from .img to HFE and back produced a byte-for-byte identical result.

    The Acid Test

    The final test of the HFE pipeline started with the Sequencer OS disk: an 800K image containing every file type the SD-1 supports: Thirteen OneProgram files. Eleven SixPrograms banks. A ThirtyPrograms bank, eight SixtyPrograms banks, four TwentyPresets banks, eight sequence files of various sizes, and the sequencer OS binary itself (656,384 bytes!), totaling forty-nine files, and leaving just five free blocks.

    The disk was encoded to HFE and loaded into the emulator. Success! The emulator accepted the disk, an everything was present. I selected the sequencer OS, hit load, and it was loaded successfully. A previously-loaded sound bank in emulator memory contained a program named GREASE-PLUS, definitely not one already on the disk. I saved it to the HFE disk, and it wrote successfully.

    We decoded the modified HFE file to an IMG and listed the contents: fifty files. Three free blocks. GREASE-PLUS in disk slot 13, a OneProgram file, two blocks. Complete success!

    Future Plans

    Now that this is done, I plan to release it on GitHub as a library. If I get around to writing the pretty GUI, I will probably see if I can sell that, because why not? The Giebler disk utilities still sell for $60!

    At any rate, the CLI will be out there and should work, for anyone who wants to build it themselves.

  • Brachytheraphy, Pluvicto, and decay curves

    Earlier in the year, I had LDR (low-dose) brachytherapy treatment for prostate cancer. The way it works is that the radiation oncologist, in concert with the urologist/surgeon, maps out where the cancer is in the prostate, and then builds up a map in 3D of exactly where to implant a set of radioactive seeds to irradiate the cancer and as little as possible of other things, like the bladder, urethra, and colon.

    The treatment can use various radioactive isotopes; in my case, we decided on Palladium-103, which has a half-life of a tiny bit less than 17 days, and decays by electron capture, which I had not previously heard of.

    One of the K-shell electrons in palladium-103 has has a chance of having nonzero probability density inside the nucleus. (Think of a big cartoon sign pointing to the nucleus that says “YOU MIGHT BE HERE” for the electron.)

    If that happens, there’s a possibility that the weak nuclear force interaction between the electron and a proton in the nucleus will convert that proton into a neutron. That transforms the atom from palladium to rhodium and emits a neutrino.

    No big deal to emit a neutrino; billions of them are constantly sleeting through us every second from the sun. But! Now the rhodium atom is missing an electron in the K-shell, so one of the existing electrons drops into that shell and now the atom has excess energy to dump. One of two things happens:

    • The “we’ve all seen this one in physics class”: the atom emits a photon (in this case a low-energy X-ray), and we’re back to normal energy. Ho hum.
    • Then there’s the “you can do that?” option: the electrons just play “hot potato” and pass around the extra energy until one is bound loosely enough to be kicked out — this is an Auger electron (named after Pierre Victor Auger, though Lise Meitner published it a year earlier — the guys get the credit again); from the radiomedical standpoint, it acts as if it were a beta particle — it’s a high-energy electron — but doesn’t come from a nuclear decay: the electron is literally handed the excess energy and sent packing with it.

    For treating cancer, both of these are good news: the Auger electron is very short range but has high interactivity with the cancer cells to put them on the Oblivion Express (okay, that’s Brian Auger, not Pierre!); the X-ray photons travel further, but aren’t as strong. This means that the radioactivity is concentrated right where it’s needed.

    But it’s not 100% absorbed.

    One of the warnings I got was to make sure that I stayed around six feet away from young children and possibly-pregnant women for the first six weeks, as those are folks who can be affected much more by even the weak radioactivity I was shedding.

    That made me wonder: just how radioactive was I, compared to when I started? Let’s make a chart!

    Fortunately palladium-103’s decay is super simple: one path to rhodium-103, which is stable, so I can use the basic decay-curve equation to figure out exactly how much Pd-103 is left over time[1].

    N(t)=N0×e(−λt)N(t) = N₀ × e^(-λt)

    This only requires us to know the decay constant λ, which we do: 16.99 days. We can plug that into a little Python program and get a nice curve:

    So the breakdown is actually pretty fast! We’re nearly at zero after 20 weeks, but because it’s an exponential curve, it’s a bit hard to read off numbers. Let’s look at that as a table:


     Weeks |  % Remaining
    ----------------------
       0.0 |     100.0000
       2.0 |      56.4887
       4.0 |      31.9097
       6.0 |      18.0254
       8.0 |      10.1823
      10.0 |       5.7519
      12.0 |       3.2492
      14.0 |       1.8354
      16.0 |       1.0368
      18.0 |       0.5857
      20.0 |       0.3308
      22.0 |       0.1869
      24.0 |       0.1056
      26.0 |       0.0596
      28.0 |       0.0337
      30.0 |       0.0190
      32.0 |       0.0107
      34.0 |       0.0061
      36.0 |       0.0034
      38.0 |       0.0019
      40.0 |       0.0011
      42.0 |       0.0006
      44.0 |       0.0003
      46.0 |       0.0002
      48.0 |       0.0001
      50.0 |       0.0001
      52.0 |       0.0000

    So at 6 weeks, the “it’s okay to stop warning people” cutoff, I’m at about 18% of the original intensity. That doesn’t give me an absolute number, but is interesting.

    I posted this on Reddit in r/ProstateCancer, just because it interested me; the mods did remove it, and fair, it’s more a curiosity than anything useful. Before it got pulled, though, I had one person ask me about how fast Pluvicto decayed — that’s Lutetium-177, and again, very fortunately a one-step-to-stable path. λ for Lu-177 is much shorter, about 6.4 days, so the curve falls off much faster:

    We’re pretty much at zero after ten weeks, which is significantly faster; the table looks like this:

     Weeks |  % Remaining
    ----------------------
       0.0 |     100.0000
       2.0 |      23.2255
       4.0 |       5.3943
       6.0 |       1.2528
       8.0 |       0.2910
      10.0 |       0.0676
      12.0 |       0.0157
      14.0 |       0.0036
      16.0 |       0.0008
      18.0 |       0.0002
      20.0 |       0.0000

    The warnings are different for Lu-177: “limit close contact (less than 3 feet) with household contacts for 2 days or with children and pregnant women for 7 days. Refrain from sexual activity for 7 days, and sleep in a separate bedroom from household contacts for 3 days, from children for 7 days, or from pregnant women for 15 days.”

    The remaining active Lu-177 is at about the same level, 20-ish percent in that amount of time, so my intuitive guess is that in terms of “radiation exposure to others”, the two are about the same.

    Pd-103 hangs around longer, but because it’s just dropping those Auger electrons and the low-energy X-rays, they don’t propagate as much, and the effect is much more localized.

    Lu-177 in Pluvicto circulates through the entire body, and binds to metastatic cancer cells there, so it makes sense that we’d want something that decayed a lot faster. (The Lu-177 decay is gamma and actual beta emission.)

    Conclusions? None really, it was simply trying to understand what was happening better, and was probably displacement activity. 🙂

    If you want to see the Python program that made the charts and tables, check out https://github.com/joemcmahon/decay_curve.

    [1] In college, I wrote my very first computer program to simulate the decay of a single U-235 atom to a stable state. I had learned about if statements and the rand() function, but not arrays. So I had a sheaf of if‘s that figuratively “ticked the clock” by one half life for each if block, moving on to the next when the random coin flip said the atom had taken another step along the decay path. It had all the structure and sophistication of a noodle.

    This was essentially a very bad and ridiculously unsophisticated Monte Carlo simulation, but in my defense, I had never written a computer program at all before, and I was extra proud I managed to make it work.

    There were a lot of long printouts of decay timelines on fanfold paper.

  • The Podcast Repair Kit

    A quick way to build, enhance, and repair your podcast episodes

    Introduction

    This all started when I was repairing a screwed-up Azuracast installation. I deinstalled and followed the “do this to remove the old volumes” instructions. That blew away all my stations and all my podcasts.

    Yes. it was early in the morning.

    Yes, I should have thought it through some more.

    No, I shouldn’t have been trying to work then.

    However. I was lucky in that I had filed my “completed” podcasts to be verified and then deleted, and I never got round to doing that, so I still had all my “lost” episodes. As ChatGPT hyperbolically put it, “[this was] the most heroic victory of procrastination I’ve ever seen.”

    Okay, sure. Anyway.

    I have fallen behind in keeping up with my podcasts. I haven’t put out podcasts for around three years or so of broadcasts. Some I had recordings of, some I only had playlists for, and some I just didn’t have at all…[I now have at least playlists for all but one! I’ll post an article on the rescue process soon.] but it seemed like I should try to catch up whatever I could, as people seem to really like to be able to listen to the show on their schedule instead of the broadcast one.

    Surveying the podcasts, I found I had the following problems to solve:

    • Some episodes were just completely missing. There were hints as to what they might have been in the radiospiral.net show announcement posts, but I didn’t have the playlists. I might be able to put together something sort of like them, but I triaged this to the bottom of the list; doing this would require a lot of time and guesswork, and the best I’d get would be another Priority 3 episode. [I figured out that the RadioSpiral now-playing bot had records of almost every show in Discord! I’ve managed to recover all but one of the playlists – the bot was dead for only one show – so all of these are in a new Priority 4, rebuild playlists in iTunes.]
    • Some episodes I had iTunes playlists for, but not recorded episodes. I can redo the voiceovers for these, regenerate the whole podcast programmatically, and splice them in. Priority 3.
    • Some episodes I have in full but they haven’t been edited yet. Some needed replaced voiceovers, some just needed some cleanup. Still would require editing time, not automation. These just need to be edited and uploaded. Priority 2.
    • Finally, I have quite a few episodes that are done and dusted and ready to go back up. Priority 1.

    Pretty good, I thought. I could get about 40 episodes back in circulation fast.

    Problems to solve – priority 1 episodes

    I looked at the priority 1 episodes and realized that there was a problem with them that I had not solved.

    I use Fission to chapterize my podcasts, because it’s pretty fast and easy to use. I can easily zoom in and out of the waveform and find “split” points that translate into podcast chapter breaks. However, I’d had a problem with it — it’s possible to add per-chapter metadata, specifically a title and a graphic according to the ID3v2.3 spec, and Fission let me do that, but the saved files seemed to have lost the data. It had worked for a bit, and then it didn’t, and I had never solved why. That was “fix #1” needed, as both priority 1 and 2 episodes should have that working.

    I resolved to skip it for the moment and get the episodes up, and figure out what was up with that later; not having the per-chapter graphics wasn’t a showstopper for those. So priority 1 episodes could go up right away, and I could come back to the lost graphics later.

    I took a little time and wrote a script that can upload a set of episodes to both Azuracast (and to the Internet Archive, so I have a backup I have to take active steps to screw up). That’s the etheric-currents-uploader subdirectory in the Podcast Repair Kit.

    It can upload to both places, with metadata and cover image included.

    # Upload one episode
python3 uploader.py \
  --identifier 20210830-round-and-round \
  --title "Etheric Currents 20210830: Round And Round" \
  --description "Description of your episode" \
  --podcast episode.mp3 \
  --cover cover.jpg

# Upload multiple with a config file:
python uploader.py --config my_episodes.json

    I won’t go into the episodes file config right now, but it made it easy to set-and-forget the upload of all 42 of my Priority 1 episodes and get back on the air.

    Priority 2: edit and upload

    I started on this process and was proceeding along with it when I hit the per-chapter metadata problem again…and solved it. I figured out how to rescue the “broken” episodes, fixing up the Priority 1’s that needed it, and readying that for the Priority 2’s that had been partially completed: chapterized, but without their final edits. See my post on figuring that out and fixing it here.

    The tools coming out of that were

    • chapter-analyzer.py: does a text dump of the contents of the ID3v2.3 data. Critical to seeing what was wrong.
    • chapter-report.py: a nice HTML version of the previous report. Heavier on the pretty, lighter on the data.
    • rescue_busted_offsets.py: The kicker. Looks at the CHAP data from a file saved by Fission and fixes the broken pointers that prevent the per-chapter graphics (which it did save) from showing! This one rescues the Priority 1’s that were already uploaded so they can be replaced with properly-chapterized ones.

    Priority 3’s: a faster, better path to recovery

    Originally, I had a dumber script that assembled raw episodes from the M3U’s. It embedded a ten-minute silence for the intro, outro, and breaks during the show, allowing me enough space to record and paste in a replacement voiceover.

    In the process of discovering how to fix the the broken chapters issue, it became clear that I might as well take advantage of the fact that I understood the problem and could solve it.

    Instead of building in dummy silences, I realized I could save a tedious manual editing step by prerecording the breaks, naming them appropriately, and then having the script that read the M3U’s pick them up and insert them instead.

    This streamlines the process of reconstituting the playlist-only shows considerably. All I have to do now is record the voiceovers I want, add them to iTunes, set the names to match the date of the episode I’m building, and run a single script.

    • build_episode_from_playlist_library.py: takes an M3U playlist (easy to export from iTunes) and reads the iTunes library to assemble a chapterized podcast file with working per-chapter metadata and graphics.

    Priority 4 episodes

    The priority 4 episodes are going to be more work, unless I can figure a way to build the M3U’s automatically. I’ve had a fair amount of success with Python scripting here so far, so another work session with Claude (the better programmer) may give me a script I can hand a list of tracks to, and get a ready-to-use M3U for Priority 3 back.

    Toil to do

    There’s going to be a certain amount of toil to take what I currently have and organize it into a work queue that I can just plow my way through; really no way to get around that, but once it’s done, it’s simply putting in the time to move items up the priority list and get them uploaded.

    Conclusions

    A whole lot of this was possible to do quickly by having ChatGPT available and asking it the right questions with data at hand to test with. The total amount of time spent to build all the rescue tools was less than a single day.

    The problem space isn’t a complicated one, and I was greatly aided by the fact that really good libraries exist in Python to do the work, and that ffmpeg is really pretty awesome when it comes to manipulating and patching together audio files.

    Do check out the repo on GitHub, and let me know what you think! (Patches/enhancements welcome!)

  • How not to repair your Azuracast

    Recently, we’ve been working on trying to build some tooling to make our Azuracast experience for our DJs and listeners a little better.

    Shooting myself in the foot: background

    We’ve been trying to work around a longstanding bug: when a new streamer connects to Azuracast, Azuracast’s Liquidsoap processing picks up the last thing the previous streamer sent as now-playing metadata, and sets it as the metadata for the new streamer.

    This makes a lot of sense if you’re coding for the situation where a streamer loses connectivity and then resumes; generally this will be short, so preserving the now-playing metadata makes the best sense.

    However, we have a rotating set of DJs who each stream for a relatively short time – our standard show is 2 hours long. So this means that if DJ One signs off, and DJ Two starts streaming without sending new metadata after they’ve connected, then DJ Two’s set seems to be a continuation of DJ One’s signoff. This is confusing, and for streamers who prefer to simply connect and stream, means that their metadata will be “wrong” for a considerable part of the show.

    Azuracast’s now-playing APIs say that we should be able to send the stream metadata any time with a call to the API:

    curl -X POST \
     --location 'https:///api/station/1/nowplaying/update' \
     --header 'Content-Type: application/json' \
     --header 'X-API-Key: xxxx:xxxx' \
     --data '{ "title" : "Live Broadcast", "artist" : ""}'

    The only problem is that on our installation running Azuracast 0.22.1, this returns a 200 and does absolutely nothing. Looking at the logs inside Azuracast, the request is being rejected because a streamer is active. I opened a bug for this, and the recommended solution was to upgrade to the current stable release, 0.23.1.

    Round 1: Upgrading Azuracast

    2025-10-19, 9 pm: I’d upgraded Azuracast before and it had been pretty much completely seamless: put up a notice, run the Azuracast updater, broadcasting stops a second, and then the new version resumes right where it left off.

    Super easy, barely an inconvenience.

    After our 7 pm show on Sunday, I noted we’d taken a nightly automated backup of our current 0.22.1 installation, and then went ahead and upgraded: broadcasting stopped a second, the UI reloaded. I had to log back in, and we were still playing the same track. Fantastic! All according to plan. I had not taken a full backup of my installation because we all know Azuracast always updates just fine.

    This was critical error #1.

    2025-10-20, 7:15 pm: The next evening, however, I tried to stream my show. All went well until about an hour and a half in, and suddenly the audio started to stutter and glitch. Badly. I took a look at the Liquidsoap logs on Azuracast and they were not pretty.

    2025/10/21 19:18:55 [clock.local_1:2] Latency is too high: we must catchup 54.91 seconds! Check if your system can process your stream fast enough (CPU usage, disk access, etc) or if your stream should be self-sync (can happen when using `input.ffmpeg`). Refer to the latency control section of the documentation for more info.

...

2025/10/21 19:18:56 [clock.local_1:2] Latency is too high: we must catchup 54.97 seconds! Check if your system can process your stream fast enough (CPU usage, disk access, etc) or if your stream should be self-sync (can happen when using `input.ffmpeg`). Refer to the latency control section of the documentation for more info.

...

2025/10/21 19:18:57 [clock.local_1:2] Latency is too high: we must catchup 55.03 seconds! Check if your system can process your stream fast enough (CPU usage, disk access, etc) or if your stream should be self-sync (can happen when using `input.ffmpeg`). Refer to the latency control section of the documentation for more info.
2025/10/21 19:18:57 [input_streamer:2] Generator max buffered length exceeded (441000 < 441180)! Dropping content..

    And so on. You can see that Liquidsoap is having a worse and worse time trying to consume my stream and send it on. I eventually stopped my show early; Liquidsoap did not recover as I expected it to, so I restarted Azuracast, and watched as the AutoDJ happily streamed away, and resolved to look at it the next day.

    No reports of problems, so I assumed it was a fluke.

    2025-10-21 7:30 pm: The next show that day, it happened again, and was just as bad. The Tuesday DJ also cut his show short.

    We had a very, very broken Azuracast, and there was an all-day streaming concert planned for Saturday, four days away.

    Round 2: rollback did not roll

    2025-10-21, 8pm: I started working right after the cancelled show, reasoning that we were indeed very much under time pressure, and that multiple restarts/crashes/reinstalls during our stations primary listener hours would be a bad idea.

    I decided to try a rollback to 0.22.1, where we’d been streaming just fine. Unfortunately, I was lacking a critical piece of information.

    When you run Azuracast’s ./docker.sh install, you must “pin” the release level you want in azuracast.env if you don’t want the most recent version. This is not documented in big bold DO THIS OR YOU WILL BE COMPLETELY SCREWED letters in the Azuracast install docs, because of course you always want the most recent stable version, why wouldn’t you?

    So I, and ChatGPT, my faithful (but unfortunately clueless about pinning versions, critical error #2: I had picked the wrong tool for the job because it gave me more answers for free) companion, embarked on getting the server fixed.

    I went through multiple iterations of “I’ve reinstalled the server and it’s upgraded itself to 0.23.1 again”. I tried multiple ways to just install 0.22.1 and leave it there.

    2025-10-21, 10:02 pm: I downloaded the code at the 0.22.1 tag and tried to run it in development mode and reinstall my automatic backup. It upgraded itself to 0.23.1.

    2025-10-21, 10:40pm: I tried building all the Docker images myself at 0.22.1 and restoring the backup. It upgraded itself.

    I tried downloading the Docker images, restoring, and just running them. It upgraded itself.

    2025-10-21, 11:55pm: I managed to dig up a full backup of our 0.22.0 install, which was around a year old. This wasn’t ideal, but it was better than nothing at all, and restored it, then tried to install 0.22.1 from source. It chugged for a long time doing the restore…and upgraded itself to 0.23.1.

    2025-10-21, 12:24 am: I then made critical error #3: I concluded that the 0.23.1 database on the database docker volume was the problem, and that I needed to deinstall Azuracast and retry the 0.22.1 install, following the documented deinstall/reinstall process. This was a bad idea, because it deleted the Docker volumes from my Azuracast install…and then erased them. So now I’d lost all my station media, all my podcasts, and all my playlists. I was very hosed. [If I had not made critical error #1 (skipping the full backup), critical error #3 would not have been a problem.]

    2025-10-21, 1:34 am: painstaking reload of the data from the old backup. It upgraded itself again.

    2025-10-21, 3:22 am: tried again, more carefully. Restore. Wait. Watch it upgrade itself again.

    2025-10-21, 4:41 am: Nothing I could think of, or that ChatGPT could think of, could fix it. We were down, hard.

    2025-10-21, 5:21 am: The rest of the team is starting to come on line. Everything was broken, I was exhausted. They chase me off to bed, and I tried to sleep.

    The rest of the team comes through

    2025-10-22, 6 am: The rest of the team is up and online. ʞu¡0ɹʞS posts a neutral “we’re down for maintenance” banner on radiospiral.net. Southwind Niehaus suggests that she can provide an alternate Azuracast server for Saturday at 0.21.0, and the team pitches in to get that server set up to be a backup.

    2025-10-22, 10 am: Mr. Spiral approves the switchover to Southwind’s server, and offers to send Gypsy Witch the tracks she needs to do her show. (She uses downloads from Azuracast to fill out her playlists.)

    2025-10-22, 10:19 am: passing out the alternate server URL to Second Life denizens starts. It is decided to not change DNS to Southwind’s server because of propagation times.

    2025-10-22, 10:27 am: plans to populate the substitute server proceed apace.

    2025-10-22, 12:16 pm: Radiospiral.net web player repointed to substitute server, but metadata is not working. Phone alerts woke me up enough that I was able to supply the right now-playing metadata URL to ʞu¡0ɹʞS .

    2025-10-22, 12:29 pm: Radiospiral.net is switched over. I update the iOS radio app’s config data on GitHub and confirm we have music but no metadata in the app; the metadata server URL was hardcoded in the released version of the app. I make a note to push out a new version with the metadata in the config file.

    2025-10-22, 1:09 pm: I am able to find the version on the App Store and make the fix.

    2025-10-22, 1:36 pm: Test version of the app up and available to beta testers.

    2025-10-22, 2:08 pm: The substitute stream is working in all the correct places in Second Life as well. We close the PI.

    I continued work on the iOS app; the real blocker was getting the screenshots right! Once that was done, I submitted the new version of the app on 10-25 and had an approval and the new version on the App Store by 10-26. Everything was working well with the substitute server, the Saturday show was successful, Southwind’s server handled the load perfectly, and kept going just fine, streaming shows and AutoDJing, while I resumed work on restoring 0.22.1.

    Actually fixing it, day 1

    I had used Claude to help verify the fixes I made to the iOS app, and it worked so much better than ChatGPT on code generation that I went ahead and subscribed at the $20/month level.

    I brought up Claude on the Azuracast server, showed it the checked-out source code repo, and asked for help solving the problem of getting to and staying on 0.22.1.

    Claude immediately told me about AZURACAST_VERSION, version pinning, and azuracast.env. [Looking back over the timeline, I wasted somewhere around 14.5 hours not knowing about that.]

    We set the AZURACAST_VERSION=0.22.1 in azuracast.env.

    Claude suggested a two-stage strategy to restore the nightly from just before the failed upgrade, and the old full backup.

    First, I checked out Azuracast again at the 0.22.1 tag and let it install itself. Claude found and fixed a couple issues that were keeping it from building.

    Once that was up and I had somewhere I could restore the files to, I first restored the old, full, backup. This got me back the media files, but not the playlists, stations, or podcasts. (It would turn out that the podcasts weren’t in that backup at all because we hadn’t started hosting them on Azuracast yet when it was taken.) That took about two hours.

    We then restored the nightly over the old backup to get the station settings back. That took only a minute, and restored the current configs and database (including playlists). I had to reset my Azuracast login password (the azuracast:account:reset-password CLI command did that).

    Because the database and the media library were not in sync, I had a lot of unassigned tracks in the library that I was going to need to get into proper playlists.

    Claude helped me build SQL queries and a small PHP program to categorize the tracks by duration

    • < 2 minutes,which are often noisy and/or disruptive
    • 2 minutes to 20 minutes (our standard AutoDJ tracks)
    • > 30 minutes, which get played on “long-play Sundays”

    and sort them into the existing playlists where they were supposed to go. The few remaining < 2-minute tracks were listened to and filed appropriately. This in total took about an hour, and the server was back in good shape.

    Day 2: Future-Proofing (~2 hours)

    We discussed what we could do to stop testing in prod. Claude suggested a blue-green deployment strategy — one known-good server at all times, so we could flip from one to the other after doing testing.

    We created /var/azuracast-staging to have somewhere to build the second server, and configured it to use ports 8000/8443 for its web interface, and the station ports on the 10xxx ports.

    The media storage is shared between prod and staging; staging has read-only access. (This is sort of useful; it doesn’t allow us to move media around on the staging server, and I may switch it to just having its own volumes that I can swap to whichever instance is currently “production”.)

    There’s now a DISASTER-RECOVERY.md document, a complete disaster recovery guide with all scenarios and an azuracast-upgrade-strategies.md that documents the blue-green deployment.

    Lessons learned

    If one is dealing with a PI with which one is not 100% a subject-matter expert, it is critical to have one available, whether a human or LLM one. I chose the wrong LLM one: as soon as I had Claude look at the configuration and told it I wanted to be running at 0.22.1 and stay there, it told me about pinning the version in azuracast.env.

    Testing in production, which is what I ended up doing with the upgrade to 0.23.1, was a bad idea. I worked with Claude to come up with a setup allowing me to run a staging Azuracast server in parallel with the production one. This lets me try things on a server that’s okay to break. It’s probably an idea to have a dev one too, but I’ll come back to that later.

    Carefully integrating full backups into the upgrade process at the correct points is critical to being able to roll back as quickly as possible. (This is carefully documented in the disaster recovery document. The recommended number of backups uses around half a terabyte of storage, but it carefully checkpoints everything along the way.)

    It’s still possible to be down for an hour or more, but not for the multiple days that resolving this took this time.

  • MP3 chapters, chapter art, and finding/fixing a bug

    True! — nervous — very, very dreadfully nervous I had been and am; but why will you say that I am mad?

    Edgar Allen Poe, The Tell-Tale Heart

    What’s a chapter?

    I started doing my Etheric Currents radio show on RadioSpiral back in 2021, and like many of the other hosts, I recorded the episodes so people who couldn’t tune in Monday nights could listen to the show whenever they had time via a podcast.

    Most podcasts just record the MP3, clean up the worst of the um’s and er’s and verbal tics (I say “all right!” waaay too much), and then publish their show as a simple MP3.

    But! MP3’s support chapters, and chapters are pretty cool. They allow your podcast to look a lot like an audiobook in podcatchers: the episode has a list of chapters, and the listener can skip around or repeat them, and the podcatchers put up the chapter info in their display.

    Rogue Amoeba’s Fission editor does a great job of creating chapters: add a “split” at the appropriate point on the timeline, and you can add the chapter metadata right there. Great UX.

    And, as shown above, podcatchers like Overcast and Apple Podcasts will show the chapters in their displays.

    Why is it important now?

    If you’re a podcaster signed up at podcastsconnect.apple.com, you probably recently got an email from Apple letting you know that they were “enhancing chapter support” in Apple-hosted podcasts. Specifically:

    • If your podcast already has chapters, nothing changes
    • If it doesn’t, then Apple’s going to use AI to “listen” to your podcast episodes and insert chapters for you.

    I’m guessing that for 90% of podcasts, that this will work pretty well, but if you are a bit obsessive about getting stuff “just so,” you will probably be happier doing this yourself.

    There are lots of other options for chapterizing your episodes; several online tools will take a list of times (mm:ss) and names, and plop all the chapters in where you want them. Fission (aforementioned) will let you do it in a nice graphical interface.

    The MP3 specification version for ID3v2.3 allows for quite a bit of information in the chapters, including per-chapter graphics. My show plays a lot of tracks from our music library, and I wanted to add the covers and track names as chapters, so a listener could see a track list and the cover for the album the track came from. (Our remit is promoting the music we play; this is another great way to remind people that the music comes from real albums by real artists, and they should pick it up to support them.)

    So I started doing that with Fission, and for the first year or so, it worked perfectly. I copied the artwork from iTunes, pasted it in, updated the metadata, and it just worked.

    I could have sworn I did this right…

    So I didn’t notice when at some point, it stopped working. The chapters were still getting created, no problem, but the extra metadata? Disappeared as soon as I saved. I had a conversation with Rogue Amoeba around that time in 2022 and it came out that yes, we should be able to do that, but the current application could not [my note: “anymore”] and it’d go on a “future enhancements” list.

    Sigh. Well.

    I mean, I get it, this is kind of niche for someone who just wants chapters; not a lot of people care about per-chapter graphics, and that’s okay. Rogue Amoeba doesn’t exactly have an infinite number of developers, and there are a lot of other things they do that I really do not want to go stale, so no shade on Rogue Amoeba at all.

    I let this go for a while. Life happened, and my unposted podcast backlog started getting…long. After a bit of a kerfuffle on the RadioSpiral Azuracast server a few weeks ago that resulted in the Docker volume that the podcasts live on getting clobbered (for anyone else running Azuracast: full backups!). I decided that I really needed to start playing catchup here: restore those, and rebuild/restore the three years of backlog. (I’ll document the process of digging myself out of that hole in another post and link it here when it’s done.)

    One of the big problems was “I want my nice chapters back.” And that’s what the rest of this post is about.

    First: prove to myself that I am not mad

    I remembered having episodes where everything worked, and I dug around in the archived episodes I had put back up, and found one. Loaded it in Fission, and skipped through the chapters. Every chapter had the title of the track and the album art. It did work.

    I tried again in Fission. I’d been using it to chapterize the Miskatonic University Podcast episodes that I’ve been editing, and that was working fine. Chapterized a current podcast of mine: chapters were there, titled fine…but no art.

    If I added it, Fission took it, and threw it away on save.

    “I felt that I breathed an atmosphere of sorrow”.

    Edgar Allen Poe

    In desperation, I asked ChatGPT: “Chapterized MP3’s: is it actually possible to have a custom graphic for each chapter?”.

    Initially, it seemed like bad news: most podcatchers don’t support the enhanced chapter stuff in the IDv32.4 spec so I was out of luck. My reply was, lookit, I have an episode right here that works in my podcatcher. Can we analyze it and see what structure it has? Whatever it does is good enough to work.

    So between the two of us, we put together a Python script using mutagen to read the good episode and dump out all the IDv3 data to see why this one was good and the others were not.

    A few minor issues with hallucinations about how mutagen works and we had a working script that dumped it all. I thought about it a bit, and had the idea: let’s compare the output from a good file to the bad one I’d just created, and see what’s different.

    The problem was right there, and easy to fix. The metadata was IDv32.3, and that the podcatchers do understand…but a little more explanation of how chapters work and what’s in them is probably useful before I start blithering about the fix and offsets and hex values.

    The basics of IDv3 data

    IDv3 data is sandwiched into the MP3 file, with offsets as pointers to each successive part and to subparts of individual bits of that data. Here’s the overview of how it’s all hooked up:

    ┌─────────────────────────────────────────────┐
│                ID3v2.3 Tag                  │
│  (this sits before the MP3 audio frames)    │
└─────────────────────────────────────────────┘
                  │
                  ▼
      ┌────────────────────────────┐
      │  CTOC (Table of Contents)  │
      │  element_id = "TOC"        │
      │  flags: TOP_LEVEL, ORDERED │
      │                            │
      │  child_element_ids:        │
      │   ["ch0", "ch1",           │
      │    "ch2", ...] → chapter   │
      │                  order     │
      └────────────────────────────┘
                  │
                  ▼
   For each child element_id listed in CTOC:
   ┌──────────────────────────────────────────────┐
   │ CHAP (Chapter frame)                         │
   │   element_id = "ch0"                         │
   │   start_time = 0                             │
   │   end_time   = 168200                        │
   │                                              │
   │   Subframes:                                 │
   │     ├── TIT2  → "Intro" (chapter title)      │
   │     └── APIC  → this chapter's artwork       │
   └──────────────────────────────────────────────┘
                  │
                  ▼
   ┌──────────────────────────────────────────────┐
   │ Next CHAP (e.g., "ch1")                      │
   │   start_time = 168200                        │
   │   end_time   = 347880                        │
   │                                              │
   │   Subframes:                                 │
   │     ├── TIT2 → "Item Title"                  │
   │     └── APIC → this chapter's (different)    │ 
   │                artwork                       │
   └──────────────────────────────────────────────┘

    So it’s a pretty simple linked list, with the subframes either containing or point to other IDv3 data. When a podcatcher gets to the CHAP data, it picks it up and displays it. Simple.

    The “good” file’s data looks like this:

    CHAP #1: 
  element_id: 'ch0' 
  start_time: 0 ms 
  end_time: 280764 ms 
  start_offset: 0 
  end_offset: 0 

  TIT2 (chapter titles): - ['Typhoon/Dymaxion Variations/Mercury Intro']
 
  APIC (per-chapter images): 
    APIC #1: 
      mime: image/jpeg 
      type: PictureType.OTHER 
      desc: '' 
      data length: 3053522 bytes 

CHAP #2: 
  element_id: 'ch1' 
  start_time: 280764 ms 
  end_time: 598804 ms 
  start_offset: 0 
  end_offset: 0 

  TIT2 (chapter titles): - ['Mountain Dreams'] 

  APIC (per-chapter images): 
    APIC #1: 
      mime: image/jpeg 
      type: PictureType.OTHER 
      desc: '' 
      data length: 192594 bytes

    Now compare the “bad” file:

    CHAP #1: 
  element_id: 'ch0' 
  start_time: 0 ms 
  end_time: 163996 ms
 
  start_offset: 4294967295 
  end_offset: 4294967295 

  TIT2 (chapter titles): - ['Etheric Currents Intro: Green'] 

  APIC (per-chapter images): 
    APIC #1: 
      mime: image/jpeg 
      type: PictureType.OTHER 
      desc: '' 
      data length: 2329950 bytes 

CHAP #2: 
  element_id: 'ch1' 
  start_time: 163996 ms 
  end_time: 483996 ms 

  start_offset: 4294967295 
  end_offset: 4294967295 

  TIT2 (chapter titles): - ['Dripping Green'] 
  APIC (per-chapter images): 
    APIC #1: 
      mime: image/jpeg 
      type: PictureType.OTHER 
      desc: '' 
      data length: 46665 bytes

    Hm. That’s different. And that number looks familiar…

    4294967295 -> 0xFFFFFFFF

    Ha! That is, according to the IDv3 spec, a NIL pointer.

    So Fission is writing the image data in, even in the “bad” file: we can see the data length is non-zero. But it’s also writing a nil “these aren’t the images you want, move along” pointer into the file, so anything reading the file says, “nope, no chapter image here, just use the base one for the file”! And that includes Fission itself when it reloads the file!

    Now that I know what’s wrong…can I fix it?

    Spoiler: YES. And it’s trivial! All I have to do is scan through the CHAP blocks, and set the start_offset and end_offset to 0.

    Take a look at rescue-busted-offsets.py in my Podcast Repair Kit repo on GitHub. Better yet: here’s output from the chapter-report.py tool (another handy little item, for visualizing your podcast episodes!)in that same repo, showing a podcast file from before and after a repair:

    I’ll have more to say about the catch-up process in general in another post.

  • Azuracast stream monitoring: the Greedy Shark

    My ear is open like a greedy shark,
    To catch the tunings of a voice divine.

    • John Keats, Woman! when I behold thee flippant, vain

    Why the Shark?

    As one of the people managing RadioSpiral, I’m the one who’s in charge of the actual audio streaming server. One of our goals is to be up and running with an active audio stream 24/7. Most of the time, this is handled by an AutoDJ bot run by the Azuracast server.

    We also have live DJs/performers who stream music to the station; when they connect, this pre-empts the AutoDJ.

    Most of the time this all works well: the AutoDJ keeps tunes spinning, the DJs cut in to do their shows, and the AutoDJ takes over again when they disconnect.

    Every once in a while, though, things don’t go as planned: network outages, DJ tech troubles, and the like. And when those happen, we go off the air.

    Perhaps unsurprisingly, there are loads of tools for monitoring computer processes — is the server up, does it serve web pages, are there any errors — but almost nothing for monitoring audio.

    So it was necessary to create one.

    What the Shark does

    The Shark exists to avoid the worst problem a radio station can have: dead air.

    It’s a Python-based monitoring system that:

    1. Actually listens to your stream using ffmpeg to capture audio samples and analyze them for silence
    2. It knows the difference between “off the air” and “off the air because the DJ’s not sending audio” by using the AzuraCast APIs to check the server status and determine what’s going on.
    3. It alerts our tech team via notifications in a private channel if the AutoDJ drops.
    4. It takes action itself to force streamers off if they stop sending audio by suspending them via the AzuraCast API.
    5. It has an associated Discord bot that lets the team check status and bring suspended streamers back without requiring a login to the AzuraCast server.

    How It Works

    To keep all this straight, I decided to use a state machine with three modes:

    • NO_STREAMER (No DJ connected): after 2 minutes of silence send a “Silence detected!” message to a private Discord channel
    • STREAMER_ACTIVE (DJ connected but silent): after 8 minutes of dead air, send a “suspension imminent” warning to the private channel, and after two more minutes, auto-suspend the streamer
    • GRACE_PERIOD: (DJ is silent, but indicates they know it): monitoring paused for 15 minutes, preventing the DJ from getting booted while they’re sweating to fix their tech issue

    This gives us some basic logic with room to expand it if we have new cases later.

    Every 60 seconds, the Shark:

    1. Captures a 10-second audio sample via FFmpeg
    2. Analyzes RMS (volume) and variance (is it just silence or a stuck tone?)
    3. Updates a consecutive silences counter
    4. Checks whether the DJ has acknowledged a silence
    5. Takes action based on current state

    This gives us reasonable granularity and keeps the amount of data crunching down.

    A paired Discord bot lets us query the state of the Shark, inform the monitor that we’re “working on it”, and unsuspend streamers who’ve been unable to or forgot to get streaming again.

    • !shark-status – Current monitoring state and suspended users
    • !working-on-it (or !woi)- Activate a 15-minute grace period
    • !sharked – List all auto-suspended DJs
    • !letin – Re-enable a suspended DJ
    • !status – Check overall Shark status

    The !woi command was the thing that turned the Shark from a somewhat annoying monitor into a useful DJ tool: if you get hung up, you know you can tell the Shark to leave you be for a bit while you fix stuff. And !letin keeps our less-technical DJs happy if they do happen to accidentally suspend themselves.

    Open Source

    The whole thing is on GitHub: https://github.com/joemcmahon/greedy-shark

    MIT licensed, fully documented, with Docker Compose setup and comprehensive tests. Got an AzuraCast server? Try it yourself!

    What’s Next

    Other stuff we could do:

    • Multi-stream support — we won’t need this, but someplace set up like SomaFM with a zillion different streams might want that. If we did that, a Web dashboard for monitoring history and status would be better than a Discord channel as the primary interface
    • SMS alerts for critical issues — most of us leave our Discord alerts on enough that a ping from the Shark will get through, even late, and we’re sufficiently spread-out geographically that someone will see the alert, but serious “we are really, really down, like hours” alters should get sent in a “dude, you need to see this” way.

    Try It Yourself!

    If you run an internet radio station with AzuraCast, you can deploy this in about 10 minutes:

    1. Clone the repo
    2. Copy .env.example to .env
    3. Fill in your AzuraCast API credentials and Discord target channel
    4. Invite the bot to your server
    5. docker compose up -d

    And the Shark will be catching the tunings of your station.

    Please let me know of any problems via the Issues on GitHub, have fun, and send patches if you do something interesting with it!

  • 18.5 and .longFormAudio

    A few months back, I put together a start at a new version of RadioSpiral, coded from the ground up to make it more lightweight and easy to use and work on: no FRadioPlayer, no Swift-Radio-Pro, just starting with the very basics to put together a SwiftUI version of the app.

    I had one station working fine, with more to do to get it to a full replacement, but I put it aside to work on other stuff.

    Pulled it back out today to run it while trying to track down what was causing odd-looking metadata from one of our streamers, built it to run on the simulator…and got a runtime error.

    Huh. Didn’t have that before.

    A little poking around and a skull session with ChatGPT about the logs, and it became clear that iOS 18.5 had tightened up the requirements for the AvAudiosession.setCategory call.

    Before iOS 18.5, setCategory was pretty loose about what options values were allowable. I need .longFormAudio to prevent iOS from terminating my app if it goes into the background for a long time, but my old options setting ([.allowBluetooth, .allowAirPlay]) was no longer valid.

    I had the choice of keeping the options and switching to .default, or sticking with .longFormAudio and dropping the options. I decided to drop them; not having them doesn’t prevent the user from changing the routing in Control Center, and with .default, my app just honors that. Since that’s what I want, I deleted the options. If you’re doing something similar in your app, here are the rules:

    AVAudioSession .longFormAudio Compatibility

    Routing PolicyAllowed Category OptionsBehavior Summary
    .default (standard)All options (.allowBluetooth, .allowAirPlay, .mixWithOthers, .duckOthers, etc.)Full programmatic routing control—Bluetooth, AirPlay, etc. can be enabled via code. Prone to getting terminated.
    .longFormAudioOnly default options — no explicit CategoryOptions allowed ( see Apple Developer docs)System assumes “long-form” (radio/podcast) playback; user must route to devices manually (e.g., via Control Center). Audio will be permitted to play in the background for long periods.

    You can still add .mixWithOthers, .duckOthers, and .interruptSpokenAudioAndMixWithOthers with .longFormAudio, and those are the ones that matter.

    For a radio app, you need the following:

    setCategory(
    _ category: .playback,
    mode: AVAudioSession.Mode,
    policy: AVAudioSession.RouteSharingPolicy,
    options: AVAudioSession.CategoryOptions = []
)
    • category: .playback implies that playing audio is central to the app. The silence switch is ignored, and audio continues in the background.
    • mode: .default is the best choice for the radio, as it works with every category, but I might try .spokenAudio, to briefly pause the audio when another app plays a short audio prompt. I think this is the mode that Overcast uses for its interruptions, where it backs up the audio just a little if another audio prompt interrupts it.
    • policy: .longFormAudio fits best here, routing to the user-selected destination for long-form audio.
    • options: for now I’m not specifying any options, as none of them seem appropriate. I might try .mixWithOthers (or make that switchable on and off); right now the “no options” version takes over all audio. Other apps like Maps use .duckOthers or .mixWithOthers to interrupt the stream; I might be able to use one of these to do the same trick of “back up a bit and resume” that Overcast does, but I think I’ll stick with the default for now.

    `So my final call is now

    session.setCategory(.playback,
                    mode: .default,
                    policy: .longFormAudio,
                    options: [])

    Posting this for reference for anyone who hits “why does my code break under 18.5?”.

  • Cursor, said the Cursor, Cursor, said the Cursor, Tension, apprehension, and dissension have begun!

    (If you don’t know the quote I’m parodying, hie thee to a bookstore and get a copy of Alfred Bester’s The Demolished Man forthwith.)

    This week’s adventure in LLM starred Cursor and a delayed refactoring I needed to do in the RadioSpiral iOS app. We are considering adding more streams in the future; our library right now ranges from the really avante-garde to hours-long drones, and those don’t really belong cheek by jowl on a single stream.

    Floating along on Thom Brennan and suddenly taking a screeching turn into Slaw will jar anybody, and we want people to enjoy the stream instead of being hauled out of a pleasant dream, fumbling to Shut That Terrible Noise Off.

    This means that we’ll need to do some work to set up and support these multiple streams. We’ve got a jump on that; Azuracast can easily support multiple stations, and the iOS app already has infrastructure built in to do it too…except that it only works for multiple streams with Shoutcast or Icecast, not Azuracast, and it definitely isn’t ready to handle switching the Azuracast websocket metadata monitor around for different stations.

    It seemed like a good idea at the time

    The original version of the app (Swift Radio Pro) handled it okay for Icecast and Shoutcast, because the stream monitoring was built right in to the radio engine (FRadioPlayer). (Great work by all concerned on each of these projects, by the way!)

    However, when we switched over to Azuracast, we started to have problems using the methods that were built in to FRadioPlayer to monitor the metadata. They just did not work.

    Unfortunately we’d already committed to the infrastructure change because our old broadcast software was a pain to maintain and required things like “regenerate the automatic playlist every couple months, or the station will go dead and just sit there until someone notices”. Which was easy to forget to do, and did not contribute to a professional impression.

    Nobody wanted to go back, least of all me since it was my bright idea, meaning that I needed to find a way fast to get the app showing metadata again.

    I did not manage fast. It was a couple months before I freed up enough time to really dive in and fix it (see multiple postings here earlier), and I invoked a quick fix that put the new metadata management in the NowPlayingViewController. It wasn’t bad, in that it (eventually) worked and we got our metadata, but when I started looking at switching stations, it started to get really messy.

    The code, in hindsight, belonged in the Station management code, and not in the NowPlaying code, which should have only been showing it and nothing else.

    I took a couple cracks at it myself, but it ended up being a complex process, and I decided this last Friday that I should see if I could get an LLM assistant to get me through this. I installed Cursor, fired it up, and started in.

    [Aside]

    I thought I should see if I could get the conversation back that I’d had with Cursor to make it easier to write this next section. I had ended the conversation at one point and started a new one, so I couldn’t easily scroll back and see what we’d both said. I thought, the UI can pull up conversations, so it’s gotta be there, right?

    Not right. I spent a couple hours with Cursor trying to reconstruct the conversation. Best we could do after a couple of hours of “no, be dumber, don’t try to filter, just get everything you can” was my side of the conversation and none of Cursor’s. Lesson learned, don’t close it out if you might want to reread it…

    [back to our regularly scheduled blog post]

    I started off by describing the change I wanted to make, and the bugs I was trying to fix: the metadata extraction, where it was, where I wanted it, and asked if we could move the code so it appeared to be using the FRadioPlayer metadata callbacks, to minimize the difference in the code. (I had forgotten that I’d switched to my own callback mechanism, which would bite me.)

    Cursor built a new class, StationsMetadataManager, but didn’t add it to the project. I did that by hand myself, and decided that the better part of valor here was to let Cursor make code changes and I would manage Xcode. We faffed around a bit getting types made public, and the code built — but no metadata. I remembered after a minute that we should be using my callbacks, and asked to move those too. (If Cursor had been a human assistant I think I would have gotten a stink-eye at that point.)

    We fiddled a bit more, and the callbacks started working again, and we were getting the updates as expected. Despite my asking Cursor to look carefully at the metadata fetch code, it didn’t realize that I had switchable trace messages and wanted to put in its own. I had it look specifically for the if debuglevel && lines, which were the traces I’d added. It remembered this time, but forgot later.

    Elapsed time: probably an hour (the recovered logs don’t have timestamps, so I can’t be certain.)

    I ran on-device for a while, and noticed that the Xcode resource graphs were showing that I was using more and more resources each time I switched stations. Cursor suggested that we might be leaking timers, and that it could add debug and watch for the timers being invalidated. I countered that I knew that LLMs sometimes had trouble counting, and perhaps we should keep a timer count instead to check for leaks.

    We did that, and it wasn’t timers; Cursor suggested maybe we were leaking metadata callbacks. A quick try with parallel tracking of resources, and yep, that was it. Cursor originally tried to be clever and have the NowPlayingViewController‘s deinit() clean up, but I pointed out that there was only one, and it never went out of scope, so it created a method to nuke all the active callbacks that a station switch would call.The resource hogging dropped back to better but not perfect levels (a streaming audio app is going to use “unreasonable” amounts of resource).

    We were still using a lot of energy, and I had an idea. The metadata server sends back a “expect the next update in N seconds” value; I proposed adding 5 seconds to that and making that the timer pop value. This would mean that most of the time we’d never pop at all — the timer would get cancelled because the metadata arrived on time — and if it didn’t, we’d get the “woops, the metadata didn’t arrive” on a reasonable schedule. We made that change and the energy use dropped again a bit more to “high, but acceptable”.

    Cursor also proposed that maybe we were updating the screen too often, and the redraws might be using energy we didn’t need to. We put together a streamlined version of == for the Equatable implementation on the stream status object to see if that would help.

    That seemed to be working until I switched to the test station that plays very short tracks (from Slaw’s Snakes and Ladders; recommended if you like your music strange and short). The metadata stopped updating for a bit, and while looking at the log I noted that we had two callbacks when I expected only one. Cursor reminded me that we also had to update the lockscreen (which I had forgotten, thanks for covering me on that!). The short tracks had ended and a longer track was playing so I tested the lockscreen. The album cover was a blank square and the metadata was the station name and description, so the metadata wasn’t getting processed.

    We looked at that for a bit, and verified that the callback wasn’t set up right. After a bit of back-and-forth, we got it, and I had full metadata on the lock screen. Looking at the log, I still saw a couple of places where the UI was getting updated with identical data, so I asked for a copy of “what we just set” so we’d be able to skip that update if it wasn’t needed.

    At this point the refactor was complete, and I had a little more than I had planned on!

    I think this was another two hours at this point.

    But, still more to do

    I figured I was on a roll. I had a constraint warning in this code since I forked it from the original, and I decided to go find it and fix it. I had tried myself before, but it turned out I was looking at the wrong screen!

    The stations screen was the one throwing the error. I tried Cursor’s recommended fix, and to quote myself “it looks like ass”. The change made the rows way too narrow and nothing lined up anymore. We reverted that, and I suggested we remove the hard constraint, and fix up the row height in the code. We tried a couple iterations of that, but it wasn’t really working well. I gave Cursor an ultimatum that if we didn’t fix it in ten minutes I was going to revert the branch; it was working okay before, it was just throwing the warning.

    I described what the visual result was and asked, is it just that the description field isn’t line-breaking? Cursor figured out that yes, that was it, widened the row a little more, and it looked good.

    We committed, I (eventually) squashed the branch, and rebased main.

    We did a little looking at branches, and eventually I decided that it was easier for me to do it by hand than talk Cursor through it. It turned out that I’d just let a lot of junk branches build up and they all needed to go. Cleaned up, the work branch rebased onto master and removed. A good day’s work.

    Another 45 minutes or so.

    Next day, a few more issues

    I played the app overnight (our station is good sleep music) and noticed that the lockscreen stopped updating after the current track finished; it never loaded the metadata for the next track.

    I suspected that the code I’d added to save battery when the app was backgrounded was the culprit, and we looked at it; turned out that I was unconditionally stopping the metadata client whenever we went into the background. We changed that to “stop it if we are not playing”, and verified it on the short-tracks station.

    I asked Cursor to update the build number to 61 (hindsight: do not ask it to do that!), which it did,and we committed again.

    Hubris strikes

    I figured, we’ve made so much work that didn’t before, that we should take on a hard one. I proposed that we change the UI so that the items on the now-playing screen shifted around when the device was rotated to landscape (or we split on the iPad): the big album cover would move left, and the controls would all move to the right.

    I am going to draw a veil over this. It did not go well, at all.

    Cursor got what I wanted, and tried to replicate the storyboard in SwiftUI, FAIL.

    Cursor couldn’t edit storyboards, and tried to walk me through it by describing it. FAIL.

    We tried to eliminate the storyboard, and do it all in code. FAIL.

    After the SwiftUI failure, Cursor started getting reluctant to make changes itself, and I kept having to ask it to make the changes it proposed.

    I eventually gave this up as a bad job, and reverted to the so-far working code.

    Two hours or so.

    Just one more thing…

    All went well the next day until a live show came on. All of a sudden, metadata updates failed completely. We were getting the fallback data that corresponded to “metadata server is down”.

    Initially, I did what any good sysadmin does: assume it was my error until proved otherwise. I checked the Azuracast logs. I verified that the metadata monitor that fills the #now-playing channel on our Discord was working. That was all okay; the non-app monitor could see the show metadata and was displaying it.

    I asked Cursor to help me trace the issue. It forgot about my debug and inserted its own, again, but it was sufficient for me to detect that the app thought the metadata server was offline, despite the fact it was capturing metadata. Cursor updated the code in the websocket parser to check that it got the is_online flag and that it was true, and the metadata started flowing again.

    I asked it to turn the built-in trace off, and revert any debugging it had added. It informed me there wasn’t any.

    After a moment, I thought, okay, it had gotten confused, and thought I meant “debugging specifically about whether the is_online flag is set”. So I said, please clean up any debug added since the last commit.

    The code is already set to use debugLevel = 0 by default, and there are no extra debug print statements present outside those controlled by the debugLevel flags. No additional debug output remains from recent changes.If you notice any specific debug output you want removed or further adjustments to verbosity, please point it out and I’ll take care of it!

    Uh, no.

    So I pointed out several things in the log that were debug statements. It removed the ones I told it to, but it stubbornly refused to believe there were any others.

    Fine. A git diff and some edits in Xcode, and I was ready to save this and make a new build, now 63. I set the build number in the Xcode UI, archvied…and Xcode cheerfully informed me I’d created build 61.

    What?

    I looked at Xcode. 63.

    Okay, I asked Cursor to set the build number before. Odds are it’s done something that made sense to it.

    Can you please set the build to 63? I’m updating the project but it doesn’t seem to be taking.

    <key>CFBundleVersion</key>
<string>63</string>
<key>LSRequiresIPhoneOS</key>

    Sigh. It wiped out the symbolic version and hard-coded it. Go look for the version that Xcode saves…okay, it’s in .pbxproj.

    Actually, can we embed CURRENT_PROJECT_VERSION from .pbxproj? that’d be a lot simpler than this.

    <key>CFBundleVersion</key>
<string>$(CURRENT_PROJECT_VERSION)</string>
<key>LSRequiresIPhoneOS</key>

    Archive…and build 63. Son of a…

    I pushed build 63 to TestFlight, committed everything, and pushed to main. Done.

    Conclusions, observations, and speculations

    • This was not a terrible idea.
      • I got the refactor done. It might have gone faster if I’d remembered to tell Cursor I had a callback mechanism in place.
      • Cursor did do well at finding things to debounce to cut down UI updates.
      • When I could see a bug, it was pretty easy to get Cursor to fix it. We went about 50-50 on who saw it and suggested a fix; I needed to be in there finding the weird ones.
    • I was probably more productive.
      • I had tried a couple times to refactor this and wasn’t successful. Cursor got me through it, so that’s a big improvement right there.
      • I had tried to fix the lockscreen metadata, specifically setting the cover, and hadn’t been able to. We managed to get that working together.
    • Some things do not translate well for LLMs.
      • Anything visual is a challenge. If I can see it and have a good guess at the problem, we can fix it fairly fast. If I don’t we end up slogging away, adding more and more debug until one of us spots the issue.
      • The attempts to update the UI for rotation were a disaster.
      • “Eventually fixing the bad constraint” went badly and slowly. Because Cursor can’t see, it really has no idea what to do to fix a UI issue.
    • Peculiar things happen.
      • When we started having issues on the UI code, Cursor seemed to become reluctant to make changes, and had to be asked to make them. I don’t know if that was because I came across as frustrated by how it was going or what, but that was weird.
      • Cursor insisting that there was no debug, and only removing it when I specifically said, “this is debug”, and refusing to believe it had done anything else. I even told it that it should check git. This was also after the point where it started not making changes unless it was directed to.

    Overall, I’d give the refactoring an A, the fixing of the lockscreen a B, and the UI work a C for the elimination of the constraint warning, and an F for the rotation that we never finished, for an average grade of a C+.

    Will I use Cursor again for Swift? Yes, but. For a pure rote exercise, it’s extremely useful. For large-scale or creative work, it’s not good at all.

  • Old man yells at cloud, redux

    I wrote about whether the cloud is a long-term safe option for your data a while back, and this article has brought me back around to thinking through that again.

    I think it’s important to realize that many of us (certainly I have) become heavily dependent on cloud services for extremely important things. Not “break the bank, on the streets” important but “huge sections of my memories not safe” important.

    I am not dissing cloud services just because they are cloud services. I am looking at them again in a more jaundiced light because they are not services in the sense of “there is something wrong, and I can talk to someone and sort this”.

    All cloud services hang a sword of Damocles over your head.

    If you somehow trigger their automated systems, they will deny you in a flash, and the chances that you will ever be able to resolve the problem are near zero, unless you are a major source of income for them (and we’re talking millions of dollars of income, not something a typical person-in-the-street can generate) or have friends in high places: either someone inside the walls who “knows a guy” or a large and public means of embarrassing them for doing something blatantly stupid via automation.

    For instance, there have been large numbers of automated account closings on Facebook of late. People have had their accounts closed in the process of opening them. Yes, this is completely nuts. No, Facebook is not doing anything. Why bother? They have plenty of users now. Who cares if a few hundred or a few thousand people lose access? (Hint: it’s the people who’ve had those accounts and trusted Facebook with their data.)

    There’s the guy who uploaded pictures of his kid’s rash for his doctor to check on Google — and lost his account.

    There have been a number of folks who lost access to their Apple IDs, resulting in major financial losses, and because they created the IDs long before two-factor and have forgotten their questions and answers…are just screwed.

    Shymala and I have had a lot of frustration with Google and Squarespace dropping the ball on her Google Workspace connected to a .studio domain. The workspace is still currently AWOL. And Squarespace is getting 5 Benjamins a year, so it’s not like this is a free account and they don’t have to care at all.

    What all these have in common is trusting an external entity to have their bests interests in mind.

    Unfortunately, even if you’re paying for it (see Squarespace), you cannot trust that a cloud provider will not arbitrarily decide that you have violated some rule – which they will not often reveal to you, because security – and you’re just done, with no options.

    So in this post, I’m going to try to look at what cloud providers I’m using and what I could do to mitigate risks with that provider.

    Google

    The biggest risk here is email. I have several different accounts, some quite old with a lot of archived email, and a few that are simply convenience accounts that I use to sign into things.

    I also have a Google workspace for mail at the pemungkah.com domain, with a couple accounts set up there.

    Risks:

    • Losing access to my primary GMail would break logins for a lot of things. I have used “Sign in with GMail” a lot. I would also lose a lot of archived mail.
      • A search in 1Password tells me I have 439 accounts associated with my primary GMail. 60-ish are “signs in with GMail” for that account.
    • Losing access for my secondary would inconvenience me more than anything else, mostly because I wouldn’t be able to receive password resets or change-of-email confirmations. Some would be harder than others to recover.
      • 29 accounts associated with my music GMail.
    • Losing access to the Google Workspace would be a relatively minor issue; I haven’t used it for a lot. I’d have to set up new logins for some of my streaming accounts.

    Probably the best thing would be to move my mail to a custom domain. I have several that would do as a base domain, and I could just set up mailboxes for the alternate accounts. A self-hosted mail service would be better; there seem to be some reasonable alternatives but they’d need to be evaluated. Using ProtonMail or Tuta with my own domain is a middle ground.

    Apple IDs

    Losing my Apple ID would be costly monetarily and emotionally.

    • Only 4 non-Apple accounts depend on my Apple ID, and I could live if I lost access to all of them.
    • All of my photos are in iCloud. If I lost access to my Apple ID, i could concievably permanently lose access to those. I have backed them up to Google Photos (yeah, not great either).
    • I’d lose access to my Apple Developer account.
    • My Find My would be broken, leading me to have to scramble to rehome all my devices. Stolen Device Protection would make that much harder to do, but I sure don’t want to turn that off.
    • I use Find My Boxes to track where stuff is in storage. If iCloud locks me out, I can’t access that data anymore. (I have written an unloader script that converts that data to HTML, so I’m less screwed than I might be there.)

    Microsoft ID

    I use this for very little. If it went away I could set up another one.

    GitHub

    A pain in the ass, but recoverable. I could create repositories somewhere else.

    Dropbox

    I need to more deeply investigate this. We use Dropbox a lot for archiving stuff, and sharing data for Shymala’s writing and art. If we lost access, there’d definitely be some problems — some of the original art has been sold, and it’d be impossible to get prints of it anymore.

    Plans

    I need to move my primary email away from GMail alone. If it’s coming to a custom domain, I can just move the mail somewhere else. Proton or Tuta seem best; self-hosting is probably even better but incurs a lot more work for me to do, plus the hosting costs, or doing it locally. (See below for thoughts on that.)

    Once I have the new mail, I need to change everything that uses GMail login first, then work through the 400 other accounts in batches. Some are simply “fuck you” accounts for the idiots who use my mail, and I can simply forget they’re there and remove them. The rest I’ll move to the new custom domain.

    I can’t completely get rid of my Apple ID without losing iCloud and Apple Developer access, so I’ll set up a new Apple ID for the custom domain, and then migrate to it. The purchases on the old account will have to stay there, so I’ll set up a new Family account with the new email as the primary and add the old Apple ID as a secondary. I will add the new account to my Apple Developer origanizatin, and move all the responsibilities over to the new account, then start purchasing the Apple Developer access from the new account.

    For GitHub, I’ll want to fork everything to the new account and close the old one. (GitHub may have a “move account”; if so I’ll use it). I will want to self-host a Git repository that I can push to as well.

    Self-hosting

    It’s possible to host most of this stuff on a server either at a colocation site or on a box literally in the house with Tailscale. Cohosted costs extra money, but has the advantage that I could put it at (say) Hetzner and outside the US, given the current political climate. In-house is max control and possibly a lesser ongoing investment but I’d incur a dependency on Tailscale. Going to have to work out the numbers and decide about this.

    Summary

    My heavy dependence on one GMail address and Google in general is not great. I should move the pemungkah.com domain to somewhere else (handwave) and use a custom email instead of GMail, and change everything over to that from the GMail address at a minimum.

    Dropbox is going to be an issue, and I still need to make sure I have a way to really for-sure back that up.

    Everything else depends on the GMail move, so I will be concentrating on that over the next few months.

  • Email handling: a rant

    Okay, this is probably preaching to the choir for anyone who reads my blog, but I’ve just gone through a supremely frustrating experience with Hilton and I’m going to vent, because I can.

    This is also partially humor, and an excuse to repeatedly mention the name of the person who triggered all this. Enjoy, Lisa Neumann of Spearfish, SD.

    The triggering incident

    Back in the day, specifically 2004, when GMail was new – so new that you had to know someone who could invite you to it – I got my name as my GMail address, because, hey, I could! 20 years on in hindsight, I should have constructed an alias and used that, because people are idiots and companies are as bad.

    So why am I ranting today in particular? Because Lisa Neumann, of Spearfish, SD (yes, I am SEOing the hell out of good ol’ Lisa here) decided that she wanted to open a Hilton Honors account. And like any sane person, she picked a random email out of the air, in this case mine, and used that. I know I always want to enter things like my home address and name, and send those to some random person on the internet who I don’t know who can then sign me up for all kinds of mailing lists or do any number of other nefarious things based on knowing my actual physical address.

    Pardon me, my sarcasm sequencer is overloading.

    Specifically, she used a variation on my GMail address. I use a version with a dot in it; she used one without. GMail allows you to add periods to your address in any combination you like, so if your GMail address is firstmiddlelast@gmail.com, then you can use first.middlelast@gmail.com, firstmiddle.last@gmail.com, first.middle.last@gmail.com, etc. etc.

    All of these are the same email address as far as Google is concerned, and this is not news. GMail has implemented addresses this way since 2004. However, large segments of the software engineer population do not seem to have figured this out, twenty years later. The Hilton engineers in particular have not, or have said, “not our problem, we just have to push signups”.

    (I pause to note that I have no idea who Lisa Neumann is, that I have never been to Spearfish, South Dakota, and that she absolutely had no reason to think using my email address was a good idea. I will also note that if I ever am in Spearfish, I know whose address to go to, and which apartment to go knock on the door of, to ask, “What exactly was going through your mind, Lisa, when you made your home address known to some random person on the internet?”.)

    In my current case, Hilton committed not one but two sins:

    • They allowed a dot-variation of a GMail address to create a new account. (I personally already had a Hilton account.)
    • They did not validate email access. So Lisa Neumann (and yes, I really hope this ends up high in the Google hits for good ol’ Lisa Neumann of Spearfish, SD) uses a random-ass email and Hilton’s software says “hyuk, okee-dokee!” and creates an account.

    Why am I ranting about this?

    Because it is stunningly common practice. People use email addresses they don’t own all the time, and companes who supposedly want valid data don’t care.

    It’s nuts. I have mentioned before on this blog that most of the different Joe McMahons that use my email are idiots, because they know damn well that they don’t own my GMail account and will never see the mails. Apparently they don’t care that the password-reset emails go to the email that they entered, and don’t control. And I use them.

    (Have I reset the password on multiple dating sites, and uploaded a bio that says, “In addition to all stuff about that, I am not very bright, because I used someone else’s email, and he has locked me out of this account. No sweet, sweet love for me!”? Yes, yes I have. Did I enjoy it? Oh, very much so.)

    The mails I get tend to be one of the following:

    • Someone has typed “joe mcmahon” (not the email address, but the name) into the “To” field, and GMail has happily filled in the most likely email, i.e., mine. If it wasn’t someone actually writing me, it’s a genuine mistake, and I don’t count that in the “what are these idiots doing” category. This most often happens when folks in Ireland are trying to send mail to a construction company (It’s Patrick there, BTW, in case someone stumbles on this while trying to figure out why he’s not getting their mail — though I do usually send a “you probably have the wrong email” to those folks, as this is only marginally their fault. Google, if they’ve never written to this person, do you think you should really do that? Maybe mention that an address was assumed, and maybe they should verify it’s right? Naaaaaah.)
    • On the other hand, we have the Joe McMahons who sign up for things. Gym memberships. Dating sites. Porn sites. Ashley Madison (a particular favorite, Joe McMahon in Australia. Don’t think I forgot.) I don’t know exactly how to judge these, though my hunch is that these are people who think Google is Magic and just putting their name and google.com will somehow get the email fairies to deliver stuff to them. Or they’re just really freaking lazy and are counting on the email not being validated. Or just don’t think about it, and when the account never gets approved because I delete the verification mail, they just assume “computers don’t work”.
    • Last we have the outright “I’m using this email and I know it’s not mine” folks, like dear old Lisa Neumann. Did I mention she’s from Spearfish, SD? It can’t be that they’re completely computer illiterate, else how would they know to use a random person’s name as an email address and expect it to work? Maybe Lisa Neumann knows/lives with/is married to a Joe McMahon in Spearfish? Can’t find one though. I’m grasping at straws here.

    But honestly, the people are not the issue here. It’s the software engineers and product managers who could keep this from happening.

    KPIs and “conversion” as a scourge on humanity

    So why would anyone implement a system guaranteed to make people hate them? Why would you implement a signup process that doesn’t care if you can send email to the person who’s signing up, when ostensibly, you want that address so you can send them email? Why would you implement a signup system that would tell me, some random dude on the internet, exactly where Lisa Neumann of Spearfish SD lives — street address and apartment number, with no recourse or warning?

    Because someone in the software development pipeline – almost certainly the product manager – has made the number of signups and/or the number of “conversions” (guest account -> permanent account) a success metric.

    It is a truism that if you make some metric critical to a system being judged as successful, people will manipulate the system and its implementation to maximize the value of that metric to the detriment of the actual goal.

    If you reward the team that closes the most bugs, teams will spam the issue tracker with trivial bugs and close them – and they’ll even add bugs to be fixed and closed.

    If you measure the success of the “conversion” page by the number of signups, then the engineers will be incentivized to “remove friction”. And the absolute easiest way to remove friction is to remove validation.

    In the case of email addresses, the dead easiest option is simply to not validate that the email is valid at all. Most engineers will not actually go that far, and allow obvious garbage to be entered as an email, but dropping the confirmation flow, or never implementing it, is a great way to get those numbers up. If any email at all, as long as it looks basically valid, is accepted, then the conversions go way up! Look, another account added! Even though the person will never be able to reset their password, or receive any notifications via email! Hey, that’s what app notifications are for anyway, and they push up our engagement KPIs! User support will figure out how to deal with the passwords!

    Sorry, need to reset the sarcasm sequencer again.

    So what is good practice?

    • If you need an email, then you validate that the person signing up can access that email. You send them an account validation link, and until they click that link, the account is not usable.
    • You follow the real world and not what the RFC says. Yes, technically, Google was incorrect to treat foobar@gmail and foo.bar@gmail as the same address, but I think their technical decision was “do we allow every combinatorial version of johnsmith to be a different account? Absolutely not, it’ll be an identity-collision nightmare.” (And when you, the implementer, allow all the combinations? Identity collision nightmare, and no one should be surprised.) So if john.smith@gmail.com has an account at your site, then someone trying to add johnsmith@gmail (Lisa) should fail.
    • Allow people to close accounts without massive manual intervention. I still have to call Hilton on the phone and try to talk someone through fixing this issue. Chat support absolutely cannot help me. Their security policy is that two accounts with different personal names can’t be merged, so I can’t merge the two accounts that use variations on the same email. And I can’t edit the name in the account that Lisa opened, so I can’t do anything to fix it myself!
    • Do not make it impossible to ever fix a bad account. I’ve had several banking accounts opened using my me.com account, and those simply cannot ever be fixed. They are set up, rightly, to require a second factor to reset the password, usually a phone number, and if it’s some dude in Vietnam who’s opened the account, I have no way to come up with his phone number, and I get to just keep marking all the bank notifications as spam, because the bank has linked his whole online identity to that email address. Even if I get hold of the bank (and good luck doing that), they can’t help me because removing the email would effectively cause the user to not exist anymore.

    I honestly think that given the unfortunate trend toward greater and greater enshittification, we’re not going to see a massive come-to-Jesus moment on not pissing off innocent bystanders, mostly because it doesn’t impact the bottom line in any significant way. I like staying at Hilton properties in general, so me boycotting them over their account handling does little to impact them, and takes something away from me.

    Unless somehow someone manages a massive fraud based on email account variations, we’re not going to see a change, and I’ll continue to block accounts for other Joes and the random Lisa Neumann (of Spearfish, SD, let’s not forget!) for the foreseeable future.

    Questions you may be asking

    • But aren’t you by implication exposing your email by saying how the dot thing works in GMail?
      • That horse is out of the barn, down the street, and out on the prairie living its best life at this point. There have been so many breaches where my email has been stolen or leaked that it doesn’t matter anymore. (I can’t think of any other way that Lisa in Spearfish (I can’t be bothered anymore) could have found it.) And GMail seems to fill it in when you type my name in the “to” field, so I’m being shafted automatically anyway.
    • Wow, shouldn’t you go touch grass or something?
      • Yes, and I totally do. It’s just that I come back to my inbox full of “WELCOME TO YOUR ACCOUNT” and “YOUR RESERVATION IS CONFIRMED” and “SexyBabe69420 sent you a wink!” messages and I might as well have not bothered.
    • Have you never done anything to people who do this?
      • Actually, beyond locking them out of the accounts they’ve opened with my identity? No. I have never cancelled a reservation, rerouted a package, or catfished someone on a dating site. I absolutely could have, but I wouldn’t respect myself for doing actual financial damage or hurting an innocent person. Messing with someone on a sex dating site? I’m only disappointing the bots.