Before the Shadow Play

Category: Programming

  • Brachytheraphy, Pluvicto, and decay curves

    Earlier in the year, I had LDR (low-dose) brachytherapy treatment for prostate cancer. The way it works is that the radiation oncologist, in concert with the urologist/surgeon, maps out where the cancer is in the prostate, and then builds up a map in 3D of exactly where to implant a set of radioactive seeds to irradiate the cancer and as little as possible of other things, like the bladder, urethra, and colon.

    The treatment can use various radioactive isotopes; in my case, we decided on Palladium-103, which has a half-life of a tiny bit less than 17 days, and decays by electron capture, which I had not previously heard of.

    One of the K-shell electrons in palladium-103 has has a chance of having nonzero probability density inside the nucleus. (Think of a big cartoon sign pointing to the nucleus that says “YOU MIGHT BE HERE” for the electron.)

    If that happens, there’s a possibility that the weak nuclear force interaction between the electron and a proton in the nucleus will convert that proton into a neutron. That transforms the atom from palladium to rhodium and emits a neutrino.

    No big deal to emit a neutrino; billions of them are constantly sleeting through us every second from the sun. But! Now the rhodium atom is missing an electron in the K-shell, so one of the existing electrons drops into that shell and now the atom has excess energy to dump. One of two things happens:

    • The “we’ve all seen this one in physics class”: the atom emits a photon (in this case a low-energy X-ray), and we’re back to normal energy. Ho hum.
    • Then there’s the “you can do that?” option: the electrons just play “hot potato” and pass around the extra energy until one is bound loosely enough to be kicked out — this is an Auger electron (named after Pierre Victor Auger, though Lise Meitner published it a year earlier — the guys get the credit again); from the radiomedical standpoint, it acts as if it were a beta particle — it’s a high-energy electron — but doesn’t come from a nuclear decay: the electron is literally handed the excess energy and sent packing with it.

    For treating cancer, both of these are good news: the Auger electron is very short range but has high interactivity with the cancer cells to put them on the Oblivion Express (okay, that’s Brian Auger, not Pierre!); the X-ray photons travel further, but aren’t as strong. This means that the radioactivity is concentrated right where it’s needed.

    But it’s not 100% absorbed.

    One of the warnings I got was to make sure that I stayed around six feet away from young children and possibly-pregnant women for the first six weeks, as those are folks who can be affected much more by even the weak radioactivity I was shedding.

    That made me wonder: just how radioactive was I, compared to when I started? Let’s make a chart!

    Fortunately palladium-103’s decay is super simple: one path to rhodium-103, which is stable, so I can use the basic decay-curve equation to figure out exactly how much Pd-103 is left over time[1].

    N(t)=N0×e(−λt)N(t) = N₀ × e^(-λt)

    This only requires us to know the decay constant λ, which we do: 16.99 days. We can plug that into a little Python program and get a nice curve:

    So the breakdown is actually pretty fast! We’re nearly at zero after 20 weeks, but because it’s an exponential curve, it’s a bit hard to read off numbers. Let’s look at that as a table:


     Weeks |  % Remaining
    ----------------------
       0.0 |     100.0000
       2.0 |      56.4887
       4.0 |      31.9097
       6.0 |      18.0254
       8.0 |      10.1823
      10.0 |       5.7519
      12.0 |       3.2492
      14.0 |       1.8354
      16.0 |       1.0368
      18.0 |       0.5857
      20.0 |       0.3308
      22.0 |       0.1869
      24.0 |       0.1056
      26.0 |       0.0596
      28.0 |       0.0337
      30.0 |       0.0190
      32.0 |       0.0107
      34.0 |       0.0061
      36.0 |       0.0034
      38.0 |       0.0019
      40.0 |       0.0011
      42.0 |       0.0006
      44.0 |       0.0003
      46.0 |       0.0002
      48.0 |       0.0001
      50.0 |       0.0001
      52.0 |       0.0000

    So at 6 weeks, the “it’s okay to stop warning people” cutoff, I’m at about 18% of the original intensity. That doesn’t give me an absolute number, but is interesting.

    I posted this on Reddit in r/ProstateCancer, just because it interested me; the mods did remove it, and fair, it’s more a curiosity than anything useful. Before it got pulled, though, I had one person ask me about how fast Pluvicto decayed — that’s Lutetium-177, and again, very fortunately a one-step-to-stable path. λ for Lu-177 is much shorter, about 6.4 days, so the curve falls off much faster:

    We’re pretty much at zero after ten weeks, which is significantly faster; the table looks like this:

     Weeks |  % Remaining
    ----------------------
       0.0 |     100.0000
       2.0 |      23.2255
       4.0 |       5.3943
       6.0 |       1.2528
       8.0 |       0.2910
      10.0 |       0.0676
      12.0 |       0.0157
      14.0 |       0.0036
      16.0 |       0.0008
      18.0 |       0.0002
      20.0 |       0.0000

    The warnings are different for Lu-177: “limit close contact (less than 3 feet) with household contacts for 2 days or with children and pregnant women for 7 days. Refrain from sexual activity for 7 days, and sleep in a separate bedroom from household contacts for 3 days, from children for 7 days, or from pregnant women for 15 days.”

    The remaining active Lu-177 is at about the same level, 20-ish percent in that amount of time, so my intuitive guess is that in terms of “radiation exposure to others”, the two are about the same.

    Pd-103 hangs around longer, but because it’s just dropping those Auger electrons and the low-energy X-rays, they don’t propagate as much, and the effect is much more localized.

    Lu-177 in Pluvicto circulates through the entire body, and binds to metastatic cancer cells there, so it makes sense that we’d want something that decayed a lot faster. (The Lu-177 decay is gamma and actual beta emission.)

    Conclusions? None really, it was simply trying to understand what was happening better, and was probably displacement activity. 🙂

    If you want to see the Python program that made the charts and tables, check out https://github.com/joemcmahon/decay_curve.

    [1] In college, I wrote my very first computer program to simulate the decay of a single U-235 atom to a stable state. I had learned about if statements and the rand() function, but not arrays. So I had a sheaf of if‘s that figuratively “ticked the clock” by one half life for each if block, moving on to the next when the random coin flip said the atom had taken another step along the decay path. It had all the structure and sophistication of a noodle.

    This was essentially a very bad and ridiculously unsophisticated Monte Carlo simulation, but in my defense, I had never written a computer program at all before, and I was extra proud I managed to make it work.

    There were a lot of long printouts of decay timelines on fanfold paper.

  • The Podcast Repair Kit

    A quick way to build, enhance, and repair your podcast episodes

    Introduction

    This all started when I was repairing a screwed-up Azuracast installation. I deinstalled and followed the “do this to remove the old volumes” instructions. That blew away all my stations and all my podcasts.

    Yes. it was early in the morning.

    Yes, I should have thought it through some more.

    No, I shouldn’t have been trying to work then.

    However. I was lucky in that I had filed my “completed” podcasts to be verified and then deleted, and I never got round to doing that, so I still had all my “lost” episodes. As ChatGPT hyperbolically put it, “[this was] the most heroic victory of procrastination I’ve ever seen.”

    Okay, sure. Anyway.

    I have fallen behind in keeping up with my podcasts. I haven’t put out podcasts for around three years or so of broadcasts. Some I had recordings of, some I only had playlists for, and some I just didn’t have at all…[I now have at least playlists for all but one! I’ll post an article on the rescue process soon.] but it seemed like I should try to catch up whatever I could, as people seem to really like to be able to listen to the show on their schedule instead of the broadcast one.

    Surveying the podcasts, I found I had the following problems to solve:

    • Some episodes were just completely missing. There were hints as to what they might have been in the radiospiral.net show announcement posts, but I didn’t have the playlists. I might be able to put together something sort of like them, but I triaged this to the bottom of the list; doing this would require a lot of time and guesswork, and the best I’d get would be another Priority 3 episode. [I figured out that the RadioSpiral now-playing bot had records of almost every show in Discord! I’ve managed to recover all but one of the playlists – the bot was dead for only one show – so all of these are in a new Priority 4, rebuild playlists in iTunes.]
    • Some episodes I had iTunes playlists for, but not recorded episodes. I can redo the voiceovers for these, regenerate the whole podcast programmatically, and splice them in. Priority 3.
    • Some episodes I have in full but they haven’t been edited yet. Some needed replaced voiceovers, some just needed some cleanup. Still would require editing time, not automation. These just need to be edited and uploaded. Priority 2.
    • Finally, I have quite a few episodes that are done and dusted and ready to go back up. Priority 1.

    Pretty good, I thought. I could get about 40 episodes back in circulation fast.

    Problems to solve – priority 1 episodes

    I looked at the priority 1 episodes and realized that there was a problem with them that I had not solved.

    I use Fission to chapterize my podcasts, because it’s pretty fast and easy to use. I can easily zoom in and out of the waveform and find “split” points that translate into podcast chapter breaks. However, I’d had a problem with it — it’s possible to add per-chapter metadata, specifically a title and a graphic according to the ID3v2.3 spec, and Fission let me do that, but the saved files seemed to have lost the data. It had worked for a bit, and then it didn’t, and I had never solved why. That was “fix #1” needed, as both priority 1 and 2 episodes should have that working.

    I resolved to skip it for the moment and get the episodes up, and figure out what was up with that later; not having the per-chapter graphics wasn’t a showstopper for those. So priority 1 episodes could go up right away, and I could come back to the lost graphics later.

    I took a little time and wrote a script that can upload a set of episodes to both Azuracast (and to the Internet Archive, so I have a backup I have to take active steps to screw up). That’s the etheric-currents-uploader subdirectory in the Podcast Repair Kit.

    It can upload to both places, with metadata and cover image included.

    # Upload one episode
python3 uploader.py \
  --identifier 20210830-round-and-round \
  --title "Etheric Currents 20210830: Round And Round" \
  --description "Description of your episode" \
  --podcast episode.mp3 \
  --cover cover.jpg

# Upload multiple with a config file:
python uploader.py --config my_episodes.json

    I won’t go into the episodes file config right now, but it made it easy to set-and-forget the upload of all 42 of my Priority 1 episodes and get back on the air.

    Priority 2: edit and upload

    I started on this process and was proceeding along with it when I hit the per-chapter metadata problem again…and solved it. I figured out how to rescue the “broken” episodes, fixing up the Priority 1’s that needed it, and readying that for the Priority 2’s that had been partially completed: chapterized, but without their final edits. See my post on figuring that out and fixing it here.

    The tools coming out of that were

    • chapter-analyzer.py: does a text dump of the contents of the ID3v2.3 data. Critical to seeing what was wrong.
    • chapter-report.py: a nice HTML version of the previous report. Heavier on the pretty, lighter on the data.
    • rescue_busted_offsets.py: The kicker. Looks at the CHAP data from a file saved by Fission and fixes the broken pointers that prevent the per-chapter graphics (which it did save) from showing! This one rescues the Priority 1’s that were already uploaded so they can be replaced with properly-chapterized ones.

    Priority 3’s: a faster, better path to recovery

    Originally, I had a dumber script that assembled raw episodes from the M3U’s. It embedded a ten-minute silence for the intro, outro, and breaks during the show, allowing me enough space to record and paste in a replacement voiceover.

    In the process of discovering how to fix the the broken chapters issue, it became clear that I might as well take advantage of the fact that I understood the problem and could solve it.

    Instead of building in dummy silences, I realized I could save a tedious manual editing step by prerecording the breaks, naming them appropriately, and then having the script that read the M3U’s pick them up and insert them instead.

    This streamlines the process of reconstituting the playlist-only shows considerably. All I have to do now is record the voiceovers I want, add them to iTunes, set the names to match the date of the episode I’m building, and run a single script.

    • build_episode_from_playlist_library.py: takes an M3U playlist (easy to export from iTunes) and reads the iTunes library to assemble a chapterized podcast file with working per-chapter metadata and graphics.

    Priority 4 episodes

    The priority 4 episodes are going to be more work, unless I can figure a way to build the M3U’s automatically. I’ve had a fair amount of success with Python scripting here so far, so another work session with Claude (the better programmer) may give me a script I can hand a list of tracks to, and get a ready-to-use M3U for Priority 3 back.

    Toil to do

    There’s going to be a certain amount of toil to take what I currently have and organize it into a work queue that I can just plow my way through; really no way to get around that, but once it’s done, it’s simply putting in the time to move items up the priority list and get them uploaded.

    Conclusions

    A whole lot of this was possible to do quickly by having ChatGPT available and asking it the right questions with data at hand to test with. The total amount of time spent to build all the rescue tools was less than a single day.

    The problem space isn’t a complicated one, and I was greatly aided by the fact that really good libraries exist in Python to do the work, and that ffmpeg is really pretty awesome when it comes to manipulating and patching together audio files.

    Do check out the repo on GitHub, and let me know what you think! (Patches/enhancements welcome!)

  • How not to repair your Azuracast

    Recently, we’ve been working on trying to build some tooling to make our Azuracast experience for our DJs and listeners a little better.

    Shooting myself in the foot: background

    We’ve been trying to work around a longstanding bug: when a new streamer connects to Azuracast, Azuracast’s Liquidsoap processing picks up the last thing the previous streamer sent as now-playing metadata, and sets it as the metadata for the new streamer.

    This makes a lot of sense if you’re coding for the situation where a streamer loses connectivity and then resumes; generally this will be short, so preserving the now-playing metadata makes the best sense.

    However, we have a rotating set of DJs who each stream for a relatively short time – our standard show is 2 hours long. So this means that if DJ One signs off, and DJ Two starts streaming without sending new metadata after they’ve connected, then DJ Two’s set seems to be a continuation of DJ One’s signoff. This is confusing, and for streamers who prefer to simply connect and stream, means that their metadata will be “wrong” for a considerable part of the show.

    Azuracast’s now-playing APIs say that we should be able to send the stream metadata any time with a call to the API:

    curl -X POST \
     --location 'https:///api/station/1/nowplaying/update' \
     --header 'Content-Type: application/json' \
     --header 'X-API-Key: xxxx:xxxx' \
     --data '{ "title" : "Live Broadcast", "artist" : ""}'

    The only problem is that on our installation running Azuracast 0.22.1, this returns a 200 and does absolutely nothing. Looking at the logs inside Azuracast, the request is being rejected because a streamer is active. I opened a bug for this, and the recommended solution was to upgrade to the current stable release, 0.23.1.

    Round 1: Upgrading Azuracast

    2025-10-19, 9 pm: I’d upgraded Azuracast before and it had been pretty much completely seamless: put up a notice, run the Azuracast updater, broadcasting stops a second, and then the new version resumes right where it left off.

    Super easy, barely an inconvenience.

    After our 7 pm show on Sunday, I noted we’d taken a nightly automated backup of our current 0.22.1 installation, and then went ahead and upgraded: broadcasting stopped a second, the UI reloaded. I had to log back in, and we were still playing the same track. Fantastic! All according to plan. I had not taken a full backup of my installation because we all know Azuracast always updates just fine.

    This was critical error #1.

    2025-10-20, 7:15 pm: The next evening, however, I tried to stream my show. All went well until about an hour and a half in, and suddenly the audio started to stutter and glitch. Badly. I took a look at the Liquidsoap logs on Azuracast and they were not pretty.

    2025/10/21 19:18:55 [clock.local_1:2] Latency is too high: we must catchup 54.91 seconds! Check if your system can process your stream fast enough (CPU usage, disk access, etc) or if your stream should be self-sync (can happen when using `input.ffmpeg`). Refer to the latency control section of the documentation for more info.

...

2025/10/21 19:18:56 [clock.local_1:2] Latency is too high: we must catchup 54.97 seconds! Check if your system can process your stream fast enough (CPU usage, disk access, etc) or if your stream should be self-sync (can happen when using `input.ffmpeg`). Refer to the latency control section of the documentation for more info.

...

2025/10/21 19:18:57 [clock.local_1:2] Latency is too high: we must catchup 55.03 seconds! Check if your system can process your stream fast enough (CPU usage, disk access, etc) or if your stream should be self-sync (can happen when using `input.ffmpeg`). Refer to the latency control section of the documentation for more info.
2025/10/21 19:18:57 [input_streamer:2] Generator max buffered length exceeded (441000 < 441180)! Dropping content..

    And so on. You can see that Liquidsoap is having a worse and worse time trying to consume my stream and send it on. I eventually stopped my show early; Liquidsoap did not recover as I expected it to, so I restarted Azuracast, and watched as the AutoDJ happily streamed away, and resolved to look at it the next day.

    No reports of problems, so I assumed it was a fluke.

    2025-10-21 7:30 pm: The next show that day, it happened again, and was just as bad. The Tuesday DJ also cut his show short.

    We had a very, very broken Azuracast, and there was an all-day streaming concert planned for Saturday, four days away.

    Round 2: rollback did not roll

    2025-10-21, 8pm: I started working right after the cancelled show, reasoning that we were indeed very much under time pressure, and that multiple restarts/crashes/reinstalls during our stations primary listener hours would be a bad idea.

    I decided to try a rollback to 0.22.1, where we’d been streaming just fine. Unfortunately, I was lacking a critical piece of information.

    When you run Azuracast’s ./docker.sh install, you must “pin” the release level you want in azuracast.env if you don’t want the most recent version. This is not documented in big bold DO THIS OR YOU WILL BE COMPLETELY SCREWED letters in the Azuracast install docs, because of course you always want the most recent stable version, why wouldn’t you?

    So I, and ChatGPT, my faithful (but unfortunately clueless about pinning versions, critical error #2: I had picked the wrong tool for the job because it gave me more answers for free) companion, embarked on getting the server fixed.

    I went through multiple iterations of “I’ve reinstalled the server and it’s upgraded itself to 0.23.1 again”. I tried multiple ways to just install 0.22.1 and leave it there.

    2025-10-21, 10:02 pm: I downloaded the code at the 0.22.1 tag and tried to run it in development mode and reinstall my automatic backup. It upgraded itself to 0.23.1.

    2025-10-21, 10:40pm: I tried building all the Docker images myself at 0.22.1 and restoring the backup. It upgraded itself.

    I tried downloading the Docker images, restoring, and just running them. It upgraded itself.

    2025-10-21, 11:55pm: I managed to dig up a full backup of our 0.22.0 install, which was around a year old. This wasn’t ideal, but it was better than nothing at all, and restored it, then tried to install 0.22.1 from source. It chugged for a long time doing the restore…and upgraded itself to 0.23.1.

    2025-10-21, 12:24 am: I then made critical error #3: I concluded that the 0.23.1 database on the database docker volume was the problem, and that I needed to deinstall Azuracast and retry the 0.22.1 install, following the documented deinstall/reinstall process. This was a bad idea, because it deleted the Docker volumes from my Azuracast install…and then erased them. So now I’d lost all my station media, all my podcasts, and all my playlists. I was very hosed. [If I had not made critical error #1 (skipping the full backup), critical error #3 would not have been a problem.]

    2025-10-21, 1:34 am: painstaking reload of the data from the old backup. It upgraded itself again.

    2025-10-21, 3:22 am: tried again, more carefully. Restore. Wait. Watch it upgrade itself again.

    2025-10-21, 4:41 am: Nothing I could think of, or that ChatGPT could think of, could fix it. We were down, hard.

    2025-10-21, 5:21 am: The rest of the team is starting to come on line. Everything was broken, I was exhausted. They chase me off to bed, and I tried to sleep.

    The rest of the team comes through

    2025-10-22, 6 am: The rest of the team is up and online. ʞu¡0ɹʞS posts a neutral “we’re down for maintenance” banner on radiospiral.net. Southwind Niehaus suggests that she can provide an alternate Azuracast server for Saturday at 0.21.0, and the team pitches in to get that server set up to be a backup.

    2025-10-22, 10 am: Mr. Spiral approves the switchover to Southwind’s server, and offers to send Gypsy Witch the tracks she needs to do her show. (She uses downloads from Azuracast to fill out her playlists.)

    2025-10-22, 10:19 am: passing out the alternate server URL to Second Life denizens starts. It is decided to not change DNS to Southwind’s server because of propagation times.

    2025-10-22, 10:27 am: plans to populate the substitute server proceed apace.

    2025-10-22, 12:16 pm: Radiospiral.net web player repointed to substitute server, but metadata is not working. Phone alerts woke me up enough that I was able to supply the right now-playing metadata URL to ʞu¡0ɹʞS .

    2025-10-22, 12:29 pm: Radiospiral.net is switched over. I update the iOS radio app’s config data on GitHub and confirm we have music but no metadata in the app; the metadata server URL was hardcoded in the released version of the app. I make a note to push out a new version with the metadata in the config file.

    2025-10-22, 1:09 pm: I am able to find the version on the App Store and make the fix.

    2025-10-22, 1:36 pm: Test version of the app up and available to beta testers.

    2025-10-22, 2:08 pm: The substitute stream is working in all the correct places in Second Life as well. We close the PI.

    I continued work on the iOS app; the real blocker was getting the screenshots right! Once that was done, I submitted the new version of the app on 10-25 and had an approval and the new version on the App Store by 10-26. Everything was working well with the substitute server, the Saturday show was successful, Southwind’s server handled the load perfectly, and kept going just fine, streaming shows and AutoDJing, while I resumed work on restoring 0.22.1.

    Actually fixing it, day 1

    I had used Claude to help verify the fixes I made to the iOS app, and it worked so much better than ChatGPT on code generation that I went ahead and subscribed at the $20/month level.

    I brought up Claude on the Azuracast server, showed it the checked-out source code repo, and asked for help solving the problem of getting to and staying on 0.22.1.

    Claude immediately told me about AZURACAST_VERSION, version pinning, and azuracast.env. [Looking back over the timeline, I wasted somewhere around 14.5 hours not knowing about that.]

    We set the AZURACAST_VERSION=0.22.1 in azuracast.env.

    Claude suggested a two-stage strategy to restore the nightly from just before the failed upgrade, and the old full backup.

    First, I checked out Azuracast again at the 0.22.1 tag and let it install itself. Claude found and fixed a couple issues that were keeping it from building.

    Once that was up and I had somewhere I could restore the files to, I first restored the old, full, backup. This got me back the media files, but not the playlists, stations, or podcasts. (It would turn out that the podcasts weren’t in that backup at all because we hadn’t started hosting them on Azuracast yet when it was taken.) That took about two hours.

    We then restored the nightly over the old backup to get the station settings back. That took only a minute, and restored the current configs and database (including playlists). I had to reset my Azuracast login password (the azuracast:account:reset-password CLI command did that).

    Because the database and the media library were not in sync, I had a lot of unassigned tracks in the library that I was going to need to get into proper playlists.

    Claude helped me build SQL queries and a small PHP program to categorize the tracks by duration

    • < 2 minutes,which are often noisy and/or disruptive
    • 2 minutes to 20 minutes (our standard AutoDJ tracks)
    • > 30 minutes, which get played on “long-play Sundays”

    and sort them into the existing playlists where they were supposed to go. The few remaining < 2-minute tracks were listened to and filed appropriately. This in total took about an hour, and the server was back in good shape.

    Day 2: Future-Proofing (~2 hours)

    We discussed what we could do to stop testing in prod. Claude suggested a blue-green deployment strategy — one known-good server at all times, so we could flip from one to the other after doing testing.

    We created /var/azuracast-staging to have somewhere to build the second server, and configured it to use ports 8000/8443 for its web interface, and the station ports on the 10xxx ports.

    The media storage is shared between prod and staging; staging has read-only access. (This is sort of useful; it doesn’t allow us to move media around on the staging server, and I may switch it to just having its own volumes that I can swap to whichever instance is currently “production”.)

    There’s now a DISASTER-RECOVERY.md document, a complete disaster recovery guide with all scenarios and an azuracast-upgrade-strategies.md that documents the blue-green deployment.

    Lessons learned

    If one is dealing with a PI with which one is not 100% a subject-matter expert, it is critical to have one available, whether a human or LLM one. I chose the wrong LLM one: as soon as I had Claude look at the configuration and told it I wanted to be running at 0.22.1 and stay there, it told me about pinning the version in azuracast.env.

    Testing in production, which is what I ended up doing with the upgrade to 0.23.1, was a bad idea. I worked with Claude to come up with a setup allowing me to run a staging Azuracast server in parallel with the production one. This lets me try things on a server that’s okay to break. It’s probably an idea to have a dev one too, but I’ll come back to that later.

    Carefully integrating full backups into the upgrade process at the correct points is critical to being able to roll back as quickly as possible. (This is carefully documented in the disaster recovery document. The recommended number of backups uses around half a terabyte of storage, but it carefully checkpoints everything along the way.)

    It’s still possible to be down for an hour or more, but not for the multiple days that resolving this took this time.

  • MP3 chapters, chapter art, and finding/fixing a bug

    True! — nervous — very, very dreadfully nervous I had been and am; but why will you say that I am mad?

    Edgar Allen Poe, The Tell-Tale Heart

    What’s a chapter?

    I started doing my Etheric Currents radio show on RadioSpiral back in 2021, and like many of the other hosts, I recorded the episodes so people who couldn’t tune in Monday nights could listen to the show whenever they had time via a podcast.

    Most podcasts just record the MP3, clean up the worst of the um’s and er’s and verbal tics (I say “all right!” waaay too much), and then publish their show as a simple MP3.

    But! MP3’s support chapters, and chapters are pretty cool. They allow your podcast to look a lot like an audiobook in podcatchers: the episode has a list of chapters, and the listener can skip around or repeat them, and the podcatchers put up the chapter info in their display.

    Rogue Amoeba’s Fission editor does a great job of creating chapters: add a “split” at the appropriate point on the timeline, and you can add the chapter metadata right there. Great UX.

    And, as shown above, podcatchers like Overcast and Apple Podcasts will show the chapters in their displays.

    Why is it important now?

    If you’re a podcaster signed up at podcastsconnect.apple.com, you probably recently got an email from Apple letting you know that they were “enhancing chapter support” in Apple-hosted podcasts. Specifically:

    • If your podcast already has chapters, nothing changes
    • If it doesn’t, then Apple’s going to use AI to “listen” to your podcast episodes and insert chapters for you.

    I’m guessing that for 90% of podcasts, that this will work pretty well, but if you are a bit obsessive about getting stuff “just so,” you will probably be happier doing this yourself.

    There are lots of other options for chapterizing your episodes; several online tools will take a list of times (mm:ss) and names, and plop all the chapters in where you want them. Fission (aforementioned) will let you do it in a nice graphical interface.

    The MP3 specification version for ID3v2.3 allows for quite a bit of information in the chapters, including per-chapter graphics. My show plays a lot of tracks from our music library, and I wanted to add the covers and track names as chapters, so a listener could see a track list and the cover for the album the track came from. (Our remit is promoting the music we play; this is another great way to remind people that the music comes from real albums by real artists, and they should pick it up to support them.)

    So I started doing that with Fission, and for the first year or so, it worked perfectly. I copied the artwork from iTunes, pasted it in, updated the metadata, and it just worked.

    I could have sworn I did this right…

    So I didn’t notice when at some point, it stopped working. The chapters were still getting created, no problem, but the extra metadata? Disappeared as soon as I saved. I had a conversation with Rogue Amoeba around that time in 2022 and it came out that yes, we should be able to do that, but the current application could not [my note: “anymore”] and it’d go on a “future enhancements” list.

    Sigh. Well.

    I mean, I get it, this is kind of niche for someone who just wants chapters; not a lot of people care about per-chapter graphics, and that’s okay. Rogue Amoeba doesn’t exactly have an infinite number of developers, and there are a lot of other things they do that I really do not want to go stale, so no shade on Rogue Amoeba at all.

    I let this go for a while. Life happened, and my unposted podcast backlog started getting…long. After a bit of a kerfuffle on the RadioSpiral Azuracast server a few weeks ago that resulted in the Docker volume that the podcasts live on getting clobbered (for anyone else running Azuracast: full backups!). I decided that I really needed to start playing catchup here: restore those, and rebuild/restore the three years of backlog. (I’ll document the process of digging myself out of that hole in another post and link it here when it’s done.)

    One of the big problems was “I want my nice chapters back.” And that’s what the rest of this post is about.

    First: prove to myself that I am not mad

    I remembered having episodes where everything worked, and I dug around in the archived episodes I had put back up, and found one. Loaded it in Fission, and skipped through the chapters. Every chapter had the title of the track and the album art. It did work.

    I tried again in Fission. I’d been using it to chapterize the Miskatonic University Podcast episodes that I’ve been editing, and that was working fine. Chapterized a current podcast of mine: chapters were there, titled fine…but no art.

    If I added it, Fission took it, and threw it away on save.

    “I felt that I breathed an atmosphere of sorrow”.

    Edgar Allen Poe

    In desperation, I asked ChatGPT: “Chapterized MP3’s: is it actually possible to have a custom graphic for each chapter?”.

    Initially, it seemed like bad news: most podcatchers don’t support the enhanced chapter stuff in the IDv32.4 spec so I was out of luck. My reply was, lookit, I have an episode right here that works in my podcatcher. Can we analyze it and see what structure it has? Whatever it does is good enough to work.

    So between the two of us, we put together a Python script using mutagen to read the good episode and dump out all the IDv3 data to see why this one was good and the others were not.

    A few minor issues with hallucinations about how mutagen works and we had a working script that dumped it all. I thought about it a bit, and had the idea: let’s compare the output from a good file to the bad one I’d just created, and see what’s different.

    The problem was right there, and easy to fix. The metadata was IDv32.3, and that the podcatchers do understand…but a little more explanation of how chapters work and what’s in them is probably useful before I start blithering about the fix and offsets and hex values.

    The basics of IDv3 data

    IDv3 data is sandwiched into the MP3 file, with offsets as pointers to each successive part and to subparts of individual bits of that data. Here’s the overview of how it’s all hooked up:

    ┌─────────────────────────────────────────────┐
│                ID3v2.3 Tag                  │
│  (this sits before the MP3 audio frames)    │
└─────────────────────────────────────────────┘
                  │
                  ▼
      ┌────────────────────────────┐
      │  CTOC (Table of Contents)  │
      │  element_id = "TOC"        │
      │  flags: TOP_LEVEL, ORDERED │
      │                            │
      │  child_element_ids:        │
      │   ["ch0", "ch1",           │
      │    "ch2", ...] → chapter   │
      │                  order     │
      └────────────────────────────┘
                  │
                  ▼
   For each child element_id listed in CTOC:
   ┌──────────────────────────────────────────────┐
   │ CHAP (Chapter frame)                         │
   │   element_id = "ch0"                         │
   │   start_time = 0                             │
   │   end_time   = 168200                        │
   │                                              │
   │   Subframes:                                 │
   │     ├── TIT2  → "Intro" (chapter title)      │
   │     └── APIC  → this chapter's artwork       │
   └──────────────────────────────────────────────┘
                  │
                  ▼
   ┌──────────────────────────────────────────────┐
   │ Next CHAP (e.g., "ch1")                      │
   │   start_time = 168200                        │
   │   end_time   = 347880                        │
   │                                              │
   │   Subframes:                                 │
   │     ├── TIT2 → "Item Title"                  │
   │     └── APIC → this chapter's (different)    │ 
   │                artwork                       │
   └──────────────────────────────────────────────┘

    So it’s a pretty simple linked list, with the subframes either containing or point to other IDv3 data. When a podcatcher gets to the CHAP data, it picks it up and displays it. Simple.

    The “good” file’s data looks like this:

    CHAP #1: 
  element_id: 'ch0' 
  start_time: 0 ms 
  end_time: 280764 ms 
  start_offset: 0 
  end_offset: 0 

  TIT2 (chapter titles): - ['Typhoon/Dymaxion Variations/Mercury Intro']
 
  APIC (per-chapter images): 
    APIC #1: 
      mime: image/jpeg 
      type: PictureType.OTHER 
      desc: '' 
      data length: 3053522 bytes 

CHAP #2: 
  element_id: 'ch1' 
  start_time: 280764 ms 
  end_time: 598804 ms 
  start_offset: 0 
  end_offset: 0 

  TIT2 (chapter titles): - ['Mountain Dreams'] 

  APIC (per-chapter images): 
    APIC #1: 
      mime: image/jpeg 
      type: PictureType.OTHER 
      desc: '' 
      data length: 192594 bytes

    Now compare the “bad” file:

    CHAP #1: 
  element_id: 'ch0' 
  start_time: 0 ms 
  end_time: 163996 ms
 
  start_offset: 4294967295 
  end_offset: 4294967295 

  TIT2 (chapter titles): - ['Etheric Currents Intro: Green'] 

  APIC (per-chapter images): 
    APIC #1: 
      mime: image/jpeg 
      type: PictureType.OTHER 
      desc: '' 
      data length: 2329950 bytes 

CHAP #2: 
  element_id: 'ch1' 
  start_time: 163996 ms 
  end_time: 483996 ms 

  start_offset: 4294967295 
  end_offset: 4294967295 

  TIT2 (chapter titles): - ['Dripping Green'] 
  APIC (per-chapter images): 
    APIC #1: 
      mime: image/jpeg 
      type: PictureType.OTHER 
      desc: '' 
      data length: 46665 bytes

    Hm. That’s different. And that number looks familiar…

    4294967295 -> 0xFFFFFFFF

    Ha! That is, according to the IDv3 spec, a NIL pointer.

    So Fission is writing the image data in, even in the “bad” file: we can see the data length is non-zero. But it’s also writing a nil “these aren’t the images you want, move along” pointer into the file, so anything reading the file says, “nope, no chapter image here, just use the base one for the file”! And that includes Fission itself when it reloads the file!

    Now that I know what’s wrong…can I fix it?

    Spoiler: YES. And it’s trivial! All I have to do is scan through the CHAP blocks, and set the start_offset and end_offset to 0.

    Take a look at rescue-busted-offsets.py in my Podcast Repair Kit repo on GitHub. Better yet: here’s output from the chapter-report.py tool (another handy little item, for visualizing your podcast episodes!)in that same repo, showing a podcast file from before and after a repair:

    I’ll have more to say about the catch-up process in general in another post.

  • Azuracast stream monitoring: the Greedy Shark

    My ear is open like a greedy shark,
    To catch the tunings of a voice divine.

    • John Keats, Woman! when I behold thee flippant, vain

    Why the Shark?

    As one of the people managing RadioSpiral, I’m the one who’s in charge of the actual audio streaming server. One of our goals is to be up and running with an active audio stream 24/7. Most of the time, this is handled by an AutoDJ bot run by the Azuracast server.

    We also have live DJs/performers who stream music to the station; when they connect, this pre-empts the AutoDJ.

    Most of the time this all works well: the AutoDJ keeps tunes spinning, the DJs cut in to do their shows, and the AutoDJ takes over again when they disconnect.

    Every once in a while, though, things don’t go as planned: network outages, DJ tech troubles, and the like. And when those happen, we go off the air.

    Perhaps unsurprisingly, there are loads of tools for monitoring computer processes — is the server up, does it serve web pages, are there any errors — but almost nothing for monitoring audio.

    So it was necessary to create one.

    What the Shark does

    The Shark exists to avoid the worst problem a radio station can have: dead air.

    It’s a Python-based monitoring system that:

    1. Actually listens to your stream using ffmpeg to capture audio samples and analyze them for silence
    2. It knows the difference between “off the air” and “off the air because the DJ’s not sending audio” by using the AzuraCast APIs to check the server status and determine what’s going on.
    3. It alerts our tech team via notifications in a private channel if the AutoDJ drops.
    4. It takes action itself to force streamers off if they stop sending audio by suspending them via the AzuraCast API.
    5. It has an associated Discord bot that lets the team check status and bring suspended streamers back without requiring a login to the AzuraCast server.

    How It Works

    To keep all this straight, I decided to use a state machine with three modes:

    • NO_STREAMER (No DJ connected): after 2 minutes of silence send a “Silence detected!” message to a private Discord channel
    • STREAMER_ACTIVE (DJ connected but silent): after 8 minutes of dead air, send a “suspension imminent” warning to the private channel, and after two more minutes, auto-suspend the streamer
    • GRACE_PERIOD: (DJ is silent, but indicates they know it): monitoring paused for 15 minutes, preventing the DJ from getting booted while they’re sweating to fix their tech issue

    This gives us some basic logic with room to expand it if we have new cases later.

    Every 60 seconds, the Shark:

    1. Captures a 10-second audio sample via FFmpeg
    2. Analyzes RMS (volume) and variance (is it just silence or a stuck tone?)
    3. Updates a consecutive silences counter
    4. Checks whether the DJ has acknowledged a silence
    5. Takes action based on current state

    This gives us reasonable granularity and keeps the amount of data crunching down.

    A paired Discord bot lets us query the state of the Shark, inform the monitor that we’re “working on it”, and unsuspend streamers who’ve been unable to or forgot to get streaming again.

    • !shark-status – Current monitoring state and suspended users
    • !working-on-it (or !woi)- Activate a 15-minute grace period
    • !sharked – List all auto-suspended DJs
    • !letin – Re-enable a suspended DJ
    • !status – Check overall Shark status

    The !woi command was the thing that turned the Shark from a somewhat annoying monitor into a useful DJ tool: if you get hung up, you know you can tell the Shark to leave you be for a bit while you fix stuff. And !letin keeps our less-technical DJs happy if they do happen to accidentally suspend themselves.

    Open Source

    The whole thing is on GitHub: https://github.com/joemcmahon/greedy-shark

    MIT licensed, fully documented, with Docker Compose setup and comprehensive tests. Got an AzuraCast server? Try it yourself!

    What’s Next

    Other stuff we could do:

    • Multi-stream support — we won’t need this, but someplace set up like SomaFM with a zillion different streams might want that. If we did that, a Web dashboard for monitoring history and status would be better than a Discord channel as the primary interface
    • SMS alerts for critical issues — most of us leave our Discord alerts on enough that a ping from the Shark will get through, even late, and we’re sufficiently spread-out geographically that someone will see the alert, but serious “we are really, really down, like hours” alters should get sent in a “dude, you need to see this” way.

    Try It Yourself!

    If you run an internet radio station with AzuraCast, you can deploy this in about 10 minutes:

    1. Clone the repo
    2. Copy .env.example to .env
    3. Fill in your AzuraCast API credentials and Discord target channel
    4. Invite the bot to your server
    5. docker compose up -d

    And the Shark will be catching the tunings of your station.

    Please let me know of any problems via the Issues on GitHub, have fun, and send patches if you do something interesting with it!

  • 18.5 and .longFormAudio

    A few months back, I put together a start at a new version of RadioSpiral, coded from the ground up to make it more lightweight and easy to use and work on: no FRadioPlayer, no Swift-Radio-Pro, just starting with the very basics to put together a SwiftUI version of the app.

    I had one station working fine, with more to do to get it to a full replacement, but I put it aside to work on other stuff.

    Pulled it back out today to run it while trying to track down what was causing odd-looking metadata from one of our streamers, built it to run on the simulator…and got a runtime error.

    Huh. Didn’t have that before.

    A little poking around and a skull session with ChatGPT about the logs, and it became clear that iOS 18.5 had tightened up the requirements for the AvAudiosession.setCategory call.

    Before iOS 18.5, setCategory was pretty loose about what options values were allowable. I need .longFormAudio to prevent iOS from terminating my app if it goes into the background for a long time, but my old options setting ([.allowBluetooth, .allowAirPlay]) was no longer valid.

    I had the choice of keeping the options and switching to .default, or sticking with .longFormAudio and dropping the options. I decided to drop them; not having them doesn’t prevent the user from changing the routing in Control Center, and with .default, my app just honors that. Since that’s what I want, I deleted the options. If you’re doing something similar in your app, here are the rules:

    AVAudioSession .longFormAudio Compatibility

    Routing PolicyAllowed Category OptionsBehavior Summary
    .default (standard)All options (.allowBluetooth, .allowAirPlay, .mixWithOthers, .duckOthers, etc.)Full programmatic routing control—Bluetooth, AirPlay, etc. can be enabled via code. Prone to getting terminated.
    .longFormAudioOnly default options — no explicit CategoryOptions allowed ( see Apple Developer docs)System assumes “long-form” (radio/podcast) playback; user must route to devices manually (e.g., via Control Center). Audio will be permitted to play in the background for long periods.

    You can still add .mixWithOthers, .duckOthers, and .interruptSpokenAudioAndMixWithOthers with .longFormAudio, and those are the ones that matter.

    For a radio app, you need the following:

    setCategory(
    _ category: .playback,
    mode: AVAudioSession.Mode,
    policy: AVAudioSession.RouteSharingPolicy,
    options: AVAudioSession.CategoryOptions = []
)
    • category: .playback implies that playing audio is central to the app. The silence switch is ignored, and audio continues in the background.
    • mode: .default is the best choice for the radio, as it works with every category, but I might try .spokenAudio, to briefly pause the audio when another app plays a short audio prompt. I think this is the mode that Overcast uses for its interruptions, where it backs up the audio just a little if another audio prompt interrupts it.
    • policy: .longFormAudio fits best here, routing to the user-selected destination for long-form audio.
    • options: for now I’m not specifying any options, as none of them seem appropriate. I might try .mixWithOthers (or make that switchable on and off); right now the “no options” version takes over all audio. Other apps like Maps use .duckOthers or .mixWithOthers to interrupt the stream; I might be able to use one of these to do the same trick of “back up a bit and resume” that Overcast does, but I think I’ll stick with the default for now.

    `So my final call is now

    session.setCategory(.playback,
                    mode: .default,
                    policy: .longFormAudio,
                    options: [])

    Posting this for reference for anyone who hits “why does my code break under 18.5?”.

  • Cursor, said the Cursor, Cursor, said the Cursor, Tension, apprehension, and dissension have begun!

    (If you don’t know the quote I’m parodying, hie thee to a bookstore and get a copy of Alfred Bester’s The Demolished Man forthwith.)

    This week’s adventure in LLM starred Cursor and a delayed refactoring I needed to do in the RadioSpiral iOS app. We are considering adding more streams in the future; our library right now ranges from the really avante-garde to hours-long drones, and those don’t really belong cheek by jowl on a single stream.

    Floating along on Thom Brennan and suddenly taking a screeching turn into Slaw will jar anybody, and we want people to enjoy the stream instead of being hauled out of a pleasant dream, fumbling to Shut That Terrible Noise Off.

    This means that we’ll need to do some work to set up and support these multiple streams. We’ve got a jump on that; Azuracast can easily support multiple stations, and the iOS app already has infrastructure built in to do it too…except that it only works for multiple streams with Shoutcast or Icecast, not Azuracast, and it definitely isn’t ready to handle switching the Azuracast websocket metadata monitor around for different stations.

    It seemed like a good idea at the time

    The original version of the app (Swift Radio Pro) handled it okay for Icecast and Shoutcast, because the stream monitoring was built right in to the radio engine (FRadioPlayer). (Great work by all concerned on each of these projects, by the way!)

    However, when we switched over to Azuracast, we started to have problems using the methods that were built in to FRadioPlayer to monitor the metadata. They just did not work.

    Unfortunately we’d already committed to the infrastructure change because our old broadcast software was a pain to maintain and required things like “regenerate the automatic playlist every couple months, or the station will go dead and just sit there until someone notices”. Which was easy to forget to do, and did not contribute to a professional impression.

    Nobody wanted to go back, least of all me since it was my bright idea, meaning that I needed to find a way fast to get the app showing metadata again.

    I did not manage fast. It was a couple months before I freed up enough time to really dive in and fix it (see multiple postings here earlier), and I invoked a quick fix that put the new metadata management in the NowPlayingViewController. It wasn’t bad, in that it (eventually) worked and we got our metadata, but when I started looking at switching stations, it started to get really messy.

    The code, in hindsight, belonged in the Station management code, and not in the NowPlaying code, which should have only been showing it and nothing else.

    I took a couple cracks at it myself, but it ended up being a complex process, and I decided this last Friday that I should see if I could get an LLM assistant to get me through this. I installed Cursor, fired it up, and started in.

    [Aside]

    I thought I should see if I could get the conversation back that I’d had with Cursor to make it easier to write this next section. I had ended the conversation at one point and started a new one, so I couldn’t easily scroll back and see what we’d both said. I thought, the UI can pull up conversations, so it’s gotta be there, right?

    Not right. I spent a couple hours with Cursor trying to reconstruct the conversation. Best we could do after a couple of hours of “no, be dumber, don’t try to filter, just get everything you can” was my side of the conversation and none of Cursor’s. Lesson learned, don’t close it out if you might want to reread it…

    [back to our regularly scheduled blog post]

    I started off by describing the change I wanted to make, and the bugs I was trying to fix: the metadata extraction, where it was, where I wanted it, and asked if we could move the code so it appeared to be using the FRadioPlayer metadata callbacks, to minimize the difference in the code. (I had forgotten that I’d switched to my own callback mechanism, which would bite me.)

    Cursor built a new class, StationsMetadataManager, but didn’t add it to the project. I did that by hand myself, and decided that the better part of valor here was to let Cursor make code changes and I would manage Xcode. We faffed around a bit getting types made public, and the code built — but no metadata. I remembered after a minute that we should be using my callbacks, and asked to move those too. (If Cursor had been a human assistant I think I would have gotten a stink-eye at that point.)

    We fiddled a bit more, and the callbacks started working again, and we were getting the updates as expected. Despite my asking Cursor to look carefully at the metadata fetch code, it didn’t realize that I had switchable trace messages and wanted to put in its own. I had it look specifically for the if debuglevel && lines, which were the traces I’d added. It remembered this time, but forgot later.

    Elapsed time: probably an hour (the recovered logs don’t have timestamps, so I can’t be certain.)

    I ran on-device for a while, and noticed that the Xcode resource graphs were showing that I was using more and more resources each time I switched stations. Cursor suggested that we might be leaking timers, and that it could add debug and watch for the timers being invalidated. I countered that I knew that LLMs sometimes had trouble counting, and perhaps we should keep a timer count instead to check for leaks.

    We did that, and it wasn’t timers; Cursor suggested maybe we were leaking metadata callbacks. A quick try with parallel tracking of resources, and yep, that was it. Cursor originally tried to be clever and have the NowPlayingViewController‘s deinit() clean up, but I pointed out that there was only one, and it never went out of scope, so it created a method to nuke all the active callbacks that a station switch would call.The resource hogging dropped back to better but not perfect levels (a streaming audio app is going to use “unreasonable” amounts of resource).

    We were still using a lot of energy, and I had an idea. The metadata server sends back a “expect the next update in N seconds” value; I proposed adding 5 seconds to that and making that the timer pop value. This would mean that most of the time we’d never pop at all — the timer would get cancelled because the metadata arrived on time — and if it didn’t, we’d get the “woops, the metadata didn’t arrive” on a reasonable schedule. We made that change and the energy use dropped again a bit more to “high, but acceptable”.

    Cursor also proposed that maybe we were updating the screen too often, and the redraws might be using energy we didn’t need to. We put together a streamlined version of == for the Equatable implementation on the stream status object to see if that would help.

    That seemed to be working until I switched to the test station that plays very short tracks (from Slaw’s Snakes and Ladders; recommended if you like your music strange and short). The metadata stopped updating for a bit, and while looking at the log I noted that we had two callbacks when I expected only one. Cursor reminded me that we also had to update the lockscreen (which I had forgotten, thanks for covering me on that!). The short tracks had ended and a longer track was playing so I tested the lockscreen. The album cover was a blank square and the metadata was the station name and description, so the metadata wasn’t getting processed.

    We looked at that for a bit, and verified that the callback wasn’t set up right. After a bit of back-and-forth, we got it, and I had full metadata on the lock screen. Looking at the log, I still saw a couple of places where the UI was getting updated with identical data, so I asked for a copy of “what we just set” so we’d be able to skip that update if it wasn’t needed.

    At this point the refactor was complete, and I had a little more than I had planned on!

    I think this was another two hours at this point.

    But, still more to do

    I figured I was on a roll. I had a constraint warning in this code since I forked it from the original, and I decided to go find it and fix it. I had tried myself before, but it turned out I was looking at the wrong screen!

    The stations screen was the one throwing the error. I tried Cursor’s recommended fix, and to quote myself “it looks like ass”. The change made the rows way too narrow and nothing lined up anymore. We reverted that, and I suggested we remove the hard constraint, and fix up the row height in the code. We tried a couple iterations of that, but it wasn’t really working well. I gave Cursor an ultimatum that if we didn’t fix it in ten minutes I was going to revert the branch; it was working okay before, it was just throwing the warning.

    I described what the visual result was and asked, is it just that the description field isn’t line-breaking? Cursor figured out that yes, that was it, widened the row a little more, and it looked good.

    We committed, I (eventually) squashed the branch, and rebased main.

    We did a little looking at branches, and eventually I decided that it was easier for me to do it by hand than talk Cursor through it. It turned out that I’d just let a lot of junk branches build up and they all needed to go. Cleaned up, the work branch rebased onto master and removed. A good day’s work.

    Another 45 minutes or so.

    Next day, a few more issues

    I played the app overnight (our station is good sleep music) and noticed that the lockscreen stopped updating after the current track finished; it never loaded the metadata for the next track.

    I suspected that the code I’d added to save battery when the app was backgrounded was the culprit, and we looked at it; turned out that I was unconditionally stopping the metadata client whenever we went into the background. We changed that to “stop it if we are not playing”, and verified it on the short-tracks station.

    I asked Cursor to update the build number to 61 (hindsight: do not ask it to do that!), which it did,and we committed again.

    Hubris strikes

    I figured, we’ve made so much work that didn’t before, that we should take on a hard one. I proposed that we change the UI so that the items on the now-playing screen shifted around when the device was rotated to landscape (or we split on the iPad): the big album cover would move left, and the controls would all move to the right.

    I am going to draw a veil over this. It did not go well, at all.

    Cursor got what I wanted, and tried to replicate the storyboard in SwiftUI, FAIL.

    Cursor couldn’t edit storyboards, and tried to walk me through it by describing it. FAIL.

    We tried to eliminate the storyboard, and do it all in code. FAIL.

    After the SwiftUI failure, Cursor started getting reluctant to make changes itself, and I kept having to ask it to make the changes it proposed.

    I eventually gave this up as a bad job, and reverted to the so-far working code.

    Two hours or so.

    Just one more thing…

    All went well the next day until a live show came on. All of a sudden, metadata updates failed completely. We were getting the fallback data that corresponded to “metadata server is down”.

    Initially, I did what any good sysadmin does: assume it was my error until proved otherwise. I checked the Azuracast logs. I verified that the metadata monitor that fills the #now-playing channel on our Discord was working. That was all okay; the non-app monitor could see the show metadata and was displaying it.

    I asked Cursor to help me trace the issue. It forgot about my debug and inserted its own, again, but it was sufficient for me to detect that the app thought the metadata server was offline, despite the fact it was capturing metadata. Cursor updated the code in the websocket parser to check that it got the is_online flag and that it was true, and the metadata started flowing again.

    I asked it to turn the built-in trace off, and revert any debugging it had added. It informed me there wasn’t any.

    After a moment, I thought, okay, it had gotten confused, and thought I meant “debugging specifically about whether the is_online flag is set”. So I said, please clean up any debug added since the last commit.

    The code is already set to use debugLevel = 0 by default, and there are no extra debug print statements present outside those controlled by the debugLevel flags. No additional debug output remains from recent changes.If you notice any specific debug output you want removed or further adjustments to verbosity, please point it out and I’ll take care of it!

    Uh, no.

    So I pointed out several things in the log that were debug statements. It removed the ones I told it to, but it stubbornly refused to believe there were any others.

    Fine. A git diff and some edits in Xcode, and I was ready to save this and make a new build, now 63. I set the build number in the Xcode UI, archvied…and Xcode cheerfully informed me I’d created build 61.

    What?

    I looked at Xcode. 63.

    Okay, I asked Cursor to set the build number before. Odds are it’s done something that made sense to it.

    Can you please set the build to 63? I’m updating the project but it doesn’t seem to be taking.

    <key>CFBundleVersion</key>
<string>63</string>
<key>LSRequiresIPhoneOS</key>

    Sigh. It wiped out the symbolic version and hard-coded it. Go look for the version that Xcode saves…okay, it’s in .pbxproj.

    Actually, can we embed CURRENT_PROJECT_VERSION from .pbxproj? that’d be a lot simpler than this.

    <key>CFBundleVersion</key>
<string>$(CURRENT_PROJECT_VERSION)</string>
<key>LSRequiresIPhoneOS</key>

    Archive…and build 63. Son of a…

    I pushed build 63 to TestFlight, committed everything, and pushed to main. Done.

    Conclusions, observations, and speculations

    • This was not a terrible idea.
      • I got the refactor done. It might have gone faster if I’d remembered to tell Cursor I had a callback mechanism in place.
      • Cursor did do well at finding things to debounce to cut down UI updates.
      • When I could see a bug, it was pretty easy to get Cursor to fix it. We went about 50-50 on who saw it and suggested a fix; I needed to be in there finding the weird ones.
    • I was probably more productive.
      • I had tried a couple times to refactor this and wasn’t successful. Cursor got me through it, so that’s a big improvement right there.
      • I had tried to fix the lockscreen metadata, specifically setting the cover, and hadn’t been able to. We managed to get that working together.
    • Some things do not translate well for LLMs.
      • Anything visual is a challenge. If I can see it and have a good guess at the problem, we can fix it fairly fast. If I don’t we end up slogging away, adding more and more debug until one of us spots the issue.
      • The attempts to update the UI for rotation were a disaster.
      • “Eventually fixing the bad constraint” went badly and slowly. Because Cursor can’t see, it really has no idea what to do to fix a UI issue.
    • Peculiar things happen.
      • When we started having issues on the UI code, Cursor seemed to become reluctant to make changes, and had to be asked to make them. I don’t know if that was because I came across as frustrated by how it was going or what, but that was weird.
      • Cursor insisting that there was no debug, and only removing it when I specifically said, “this is debug”, and refusing to believe it had done anything else. I even told it that it should check git. This was also after the point where it started not making changes unless it was directed to.

    Overall, I’d give the refactoring an A, the fixing of the lockscreen a B, and the UI work a C for the elimination of the constraint warning, and an F for the rotation that we never finished, for an average grade of a C+.

    Will I use Cursor again for Swift? Yes, but. For a pure rote exercise, it’s extremely useful. For large-scale or creative work, it’s not good at all.

  • Azuracast metadata redux

    Summary: all for naught, back to the original implementation, but with some guardrails

    Where we last left off, I was trying to get the LDSwiftEventSource library to play nice with iOS, and it just would not. Every way I tried to convince iOS to please let this thing run failed. Even the “cancel and restart” version was a failure.

    So I started looking at the option of a central server that would push the updates using notifications, and being completely honest, it seemed like an awful lot of work that I wasn’t all that interested in doing, and which would push the release date even further out.

    On reflection, I seemed to remember that despite it being fragile as far as staying connected, the websocket implementation was rock-solid (when it was connected). I went back to that version (thank heavens for git!) and relaunched…yeah, it’s fine. It’s fine in the background. All right, how can I make this work?

    Thinking about it for a while, I also remembered that there was a ping parameter in the connect message from Azuracast, which gave the maximum interval between messages (I’ve found in practice that this is what it means; the messages usually arrive every 15 seconds or so with a ping of 25). Since I’d already written the timer code once to force reboots of the SSE code, it seemed reasonable to leverage it like this:

    • When the server connects, we get the initial ping value when we process the first message successfully.
    • I double that value, and set a Timer that will call a method that just executes connect() again if it pops.
    • In the message processing, as soon as I get a new message, I therefore have evidence that I’m connected, so I kill the extant timer, process the message, and then set a new one.

    This loops, so each time I get a message, I tell the timer I’m fine, and then set a new one; if I ever do lose connectivity, then the timer goes off and I try reconnecting.

    This still needs a couple things:

    • The retries should be limited, and do an exponential backoff.
    • I’m of two minds as to whether I throw up an indicator that I can’t reconnect to the metadata server. On one hand, the metadata going out of sync is something I am going to all these lengths to avoid, so if I’m absolutely forced to do without it, I should probably mention that it’s no longer in sync. On the other hand, if we’ve completely lost connectivity, the music will stop, and that’s a pretty significant signal in itself. It strikes me as unlikely that I’ll be able to stream from the server but not contact Azuracast, so for now I’ll just say nothing. Right now, I fall back to showing the channel metadata, so we still see we’re on RadioSpiral, but not what’s actually playing — just like when I didn’t have a working metadata implementation at all.

    I’m running it longer-term to see how well it performs. Last night I got 4 hours without a drop on the no-timer version; I think this means that drops will be relatively infrequent, and we’ll mostly just schedule Timers and cancel them.

    Lockscreen follies

    I have also been trying to get the lock screen filled out so it looks nicer. Before I started, I had a generic lockscreen that had the station logo, name and slug line with a play/pause button and two empty “–:–” timestamps. I now have an empty image (boo) but have managed to set the track name and artist name and the play time. So some progress, some regress.

    The lockscreen setup is peculiar: you set as many of the pieces of data that you know in a struct supplied by iOS, and then call a method to commit it.

    I spent a lot of time trying to get the cover to appear and couldn’t, so I left it as the channel/station logo. [Update August 2025: I’ve managed to get Cursor to work through the mess and show all the metadata! Yay.]

  • Flutter experiences

    TL;DR: Flutter builds are as much fun as Java and Scala ones, and you spend more time screwing with the tools than you do getting anything done. I don’t think I’m going to switch, at least not now.

    As I’ve mentioned before on the blog, I maintain an iOS application for RadioSpiral’s online radio station. The app has worked well and successfully; the original codebase was Swift- Radio-Pro, which works as an iOS app and a MacOS one as well (I have been doing some infrastructure changes to support Azuracast, as previously documented on the blog.)

    We do have several, very polite, Android users who inquire from time to time if I’ve ported the radio station app to Android yet, and I have had to keep saying no, as the work to duplicate the app on Android looked daunting, and nobody is paying me for this. So I’ve been putting it off, knowing that I would have to learn something that runs on Android sooner or later if I wanted to do it at all.

    Randal Schwartz has been telling me for more than a year that I really should look at Dart and Flutter if I want to maintain something that works the same on both platforms, and I just didn’t have the spare time to learn it.

    Come the end of May 2023, and I found myself laid off, so I really had nothing but time. And I was going to need to update the app for IOS 16 anyway at that point (the last time I recompiled it, Xcode still accepted iOS 8 as a target!) and I figured now was as good a time as any to see if I could get it working multi-platform.

    I started looking around for a sample Flutter radio app, and found RadioSai. From the README, it basically does what I want, but has a bunch of other features that I don’t. I figured an app I could strip down was at least a reasonable place to start, so I checked it out of Github and started to work.

    Gearing up

    Setting up the infrastructure Installing Dart and Flutter was pretty easy: good old Homebrew let me brew install flutter to get those in place, and per instructions, I ran flutter doctor to check my installation. It let me know that I was missing the Android toolchain (no surprise there, since I hadn’t installed
    anything there yet). I downloaded the current Android Studio (Flamingo in my case), opened the .dmg, and copied it into /Applications as directed.

    Rerunning flutter doctor, it now told me that I didn’t have the most recent version of the command-line tools. I then fell into a bit of a rabbit hole. Some quick Googling told me that the command line tools should live inside Android Studio. I ferreted around in the application bundle and they were just Not There. I went back to the Android Studio site and downloaded them, and spent a fair amount of time trying to get sdkmanager into my PATH correctly. When I finally did, it cheerfully informed me that I had no Java SDK. So off to the OpenJDK site, and download JDK 20. (I tried a direct install via brew install, but strangely Java was still /usr/bin/java, and I decided rather than tracking down where the Homebrew Java went, I’d install my own where l could keep an eye on it.

    I downloaded the bin.tar.gz file and followed the installation instructions, adding the specified path to my PATH… and still didn’t have a working Java. Hm. Looking in the OpenJDK directory, the path was Contents, not jdk-18.0.1.jdk/Contents. I created the jdk-18.0.1 directory, moved Contents into it and had a working Java! Hurray! But even with dorking around further with the PATH, I still couldn’t get sdkmanager to update the command-line tools properly.

    Not that way, this way

    A little more Googling turned up a Stack Overflow post that told me to forget about installing the command-line tools myself, and to get Android Studio to do it. Following those instructions and checking all the right boxes, flutter doctor told me I had the command-line tools, but that I needed to accept some licenses. I ran the command to do that, and finally I had a working Flutter install!


    Almost.

    When I launched Android Studio and loaded my project, it failed with flutter.sdk not defined. This turned out to mean that I needed to add 

    flutter.sdk=/opt/homebrew/Caskroom/flutter/ 3.10.5/flutter

    (the location that Homebrew had used to unpack Flutter — thank you find) to local.properties. After that, Gradle twiddled its fingers a while, and declared that the app was ready. (It did want to upgrade the build, and I let it do that.)

    Build, and…

    The option 'android.enableR8' is deprecated. 
It was removed in version 7.0 of the
Android Gradle plugin. 
Please remove it from 'gradle.properties".

    Okay, I remove it.

    /Users/joemcmahon/Code/radiosai/.dart_tool/ does not exist.

    More Googling, Stack Overflow says Run Tools > Flutter > Pub Get. Doesn’t exist. Okaaaaaay.

    There’s a command line version: 

    flutter clean; flutter pub get

    Deleted dart_tool, then recreated it with package_config.json there. Right!

    Back to Android Studio, still confused about the missing menu entry, and build again. Gradle runs, downloads a ton of POMs and 

    Couldn't resolve the package 'radiosai' in 'package:radiosai/audio_service/service_locator.dart'.

    Looking one level up, in :app:compileFlutterBuildDebug, Invalid depfile: /Users/joemcmahon/ Code/radiosai/.dart_tool/flutter_build/bff84666834b820d28a58a702f2c8321/ kernel_snapshot.d.

    Let’s delete those and see if that helps…yes, but still can’t resolve
    radiosai. Okay, time for a break.

    Finally, a build!

    Another Google: I wasn’t able to resolve the package because I needed to pub get again. 

    Module was compiled with an incompatible version of Kotlin.

    The binary version of its metadata is 1.8.0, expected version is 1.6.0. Another Google. One of the build Gradle files is specifying Kotlin 1.6…it’s in /android/ build.gradle. Update that to 1.8.10, build…Kotlin plugin is being loaded, good. Couple
    warnings, still going, good. 

    BUILD SUCCESSFUL

    Nice! Now, how do I test this thing? Well, there’s Device Manager over on the right, that looks promising. There’s a “Pixel 3a” entry and a “run” button. What’s the worst that could happen?

    Starts up, I have a “running device” that’s a couple inches tall, on its home screen. Hm. Ah, float AND zoom. Cool. Now I realize I have no idea how to run an Android phone, and I don’t see the app.

    https://developer.android.com/studio/run/emulator…nope. Beginning to remember why I didn’t like working in Scala… Gradle upgrade recommended, okay, and now 

    Namespace not specified. Please specify a namespace in the module's build.gradle.

    Specified, still broken…googling…This is a known issue –
    https://github.com/ionic-team/capacitor/issues/6504

    If you are using Capacitor 4, do not upgrade to Gradle 8.


    Yeah, I remember why I stopped liking Scala. git reset to put everything back…

    Execution failed for task:gallery_saver:compileDebugKotlin'. 
> compileDebugJavaWithJavac task (current target is 1.8) and 'compileDebugKotlin' task
(current target is 17) 
jvm target compatibility should be set to the same Java version.
Consider using JVM toolchain: https://kotl.in/gradle/jvm/toolchain

    Fix android/app/build.gradle so everyone thinks we’re using Java 17, which uses a different syntax, ugh.

    Fix it again. Same for the Kotlin target too. 

    'compileDebugJavaWithJavac' task (current target is 1.8) and 'compileDebugKotlin' task (current target is 17) jvm target compatibility should be set to the same Java version.

    This is apparently actually Gradle 8 still lying around after the (incorrectly) recommended upgrade. Removing ~/ gradle to nuke from orbit. Also killing android/.gradle.


    [Aside: I am used to using git grep to find things, and it is just not finding them in this repo!]

    Cannot read the array length because "" is null

    WHAT.

    Apparently this means that Gradle 8 is still lurking. Yep, the rm ~/.gradle/* didn’t remove everything because of permissions. Yougoddabefuckingkiddingme. Sudo’ed it, relaunched with the fixes I made above. App runs!


    However it stops working after a bit with no reason indicating why. Let’s stop it and restart. Stop button did not stop it; had to quit Android Studio.

    Well. Okay. This is not promising, but let’s see the benefit of using Flutter; we’ll check out if the iOS side works. Seems a lot more straightforward, though I’m not doing much in Xcode. cd iOS, launch the simulator (important!), flutter run…and we get the Flutter demo project. Looks like the IOS version wasn’t brought over from the Android side. Why did you even do this.

    Do we all remember that I wanted something that worked on both platforms? I do. We don’t. Gah.

    So I’m putting Flutter aside, cleaning up the ton of disk space all this extra infrastructure took up, and will maybe come back to it another time.

    But for right now, the amount of work involved is absolutely not worth it because I’d have to write the damn thing from scratch anyway.

    Maybe I’ll run this through one of the LLMs and see if it can get me a common codebase as a starting point, but I am not sanguine.

    [Note from the future: my fellow DJ from RadioSpiral, DJ Cosmos, has written a Go/Flutter implementation that works great on Linux and Android, so I don’t have to do this anymore!]

  • Azuracast high-frequency updates, SSE, and iOS background processes

    A big set of learning since the last update.

    I’ve been working on getting the RadioSpiral infrastructure back up to snuff after our Azuracast streaming server upgrade. We really, really did need to do that — it just provides 90% of everything we need to run the station easily right out of the box.

    Not having to regenerate the playlists every few weeks is definitely a win, and we’re now able to easily do stuff like “long-play Sunday”, where all of the tracks are long-players of a half-hour or more.

    But there were some hitches, mostly in my stuff: the iOS app and the now-playing Discord bot. Because of reasons (read: I’m not sure why), the Icecast metadata isn’t available from the streaming server on Azuracast, especially when you’re using TLS. This breaks the display of artist and track on the iOS app, and partially breaks the icecast-monitor Node library I was using to do the now-playing bot in Discord.

    (Side note: this was all my bright idea, and I should have tested the app and bot against Azuracast before I proposed cutting over in production, but I didn’t. I’ll run any new thing in Docker first and test it better next time.)

    Azuracast to the rescue

    Fortunately, Azuracast provides excellent now-playing APIs. There a straight-up GET endpoint that returns the data, and two event-driven ones (websockets and SSE). Even a “look, just read this file, it’s there” version.

    The GET option depends on you polling the server for updates, and I didn’t like that on principle; the server is quite powerful, but I don’t want multiple copies of the app hammering it frequently to get updates, and it was inherently not going to be close to a real-time update unless I really did hammer the server.

    So that was off the table, leaving websockets and SSE, neither of which I had ever used. Woo, learning experience. I initially tried SSE in Node and didn’t have a lot of success with it, so I decided to go with websockets and see how that went.

    Pretty well actually! I was able to get a websocket client running pretty easily, so I decided to try it that way. After some conferring with ChatGPT, I put together a library that would let me start up a websocket client and run happily, waiting for updates to come in and updating the UI as I went. (I’ll talk about the adventures of parsing Azuracast metadata JSON in another post.)

    I chose to use a technique that I found in the FRadioPlayer source code, of declaring a public static variable containing an instance of the class; this let me do

    import Kingfisher
import ACWebSocketClient

client = ACWebSocketClient.shared
...
tracklabel.text = client.status.track
artistlabel.text = client.status.artist
coverImageView.kf.getImage(with:client.status.artURL)

    (Kingfisher is fantastic! Coupled with Azuracast automatically extracting the artwork from tracks and providing a URL to it, showing the right covers was trivial. FRadioPlayer uses the Apple Music cover art API to get covers, and given the, shall we say, obscure artists we play, some of the cover guesses it made were pretty funny. And sometimes really inappropriate.)

    Right. So we have metadata! Fantastic. Unfortunately, the websocket client uses URLSessionWebSocketTask to manage the connection, and that class has extremely poor error handling. It’s next to impossible to detect that you’ve lost the connection or re-establish it. So It would work for a while, and then a disconnect would happen, and the metadata would stop updating.

    Back to the drawing board. Maybe SSE will work better in Swift? I’ve written one client, maybe I can leverage the code. And yes, I could. After some searching on GitHub and trying a couple of different things, I created a new library that could do Azuracast SSE. (Thank you to LaunchDarkly and LDSwiftEventSource for making the basic implementation dead easy.)

    So close, but so far

    Unfortunately, I now hit iOS architecture issues.

    iOS really, really does not want you to run long-term background tasks, especially with the screen locked. When the screen was unlocked, the metadata updates went okay, but as soon as the screen locked, iOS started a 30-second “and what do you think you’re doing” timer, and killed the metadata monitor process.

    I tried a number of gyrations to keep it running and schedule and reschedule a background thread, but if I let it run continuously, even with all the “please just let this run, I swear I know what I need here” code, iOS would axe it within a minute or so.

    So I’ve fallen back to a solution not a lot better than polling the endpoint: when the audio starts, I start up the SSE client, and then shut it down in 3 seconds, wait 15 seconds, and then run it again. When audio stops, I shut it off and leave it off. This has so far kept iOS from nuking the app, but again, I’m polling. Yuck.

    However, we now do have metadata, and that’s better than none.

    [From the future: this just was awful. I abandoned it and went back to the websockets. New update coming soon about some optimizations to save battery.]

    On the other hand…

    On the Discord front, however, I was much more successful. I tried SSE in Node, and found the libraries wanting, so I switched over to Python and was able to use sseclient to do the heavy lifting for the SSE connection. It essentially takes an SSE URL, hooks up to the server, and then calls a callback whenever an event arrives. That was straightforward enough, and I boned up on my Python for traversing arbitrary structures — json.loads() did a nice job for me of turning the complicated JSON into nested Python data structures.

    The only hard bit was persuading Python to turn the JSON struct I needed to send into a proper query parameter. Eventually this worked:

    subs = {
        "subs": {
            f"station:{shortcode}": {"recover": True}
        }
     }

json_subs = json.dumps(subs, separators=(',', ':'))
json_subs = json_subs.replace("True", "true").replace("False", "false")
encoded_query = urllib.parse.quote(json_subs)

    I pretty quickly got the events arriving and parsed, and I was able to dump out the metadata in a print. Fab! I must almost be done!

    But no. I did have to learn yet another new thing: nonlocal in Python.

    Once I’d gotten the event and parsed it and stashed the data in an object, I needed to be able to do something with it, and the easiest way to do that was set up another callback mechanism. That looked something like this:

    client = build_sse_client(server, shortcode)
run(client, send_embed_with_image)

    The send_embed_with_image callback puts together a Discord embed (a fancy message) and posts it to our Discord via a webhook, so I don’t have to write any async code. The SSE client updates every fifteen seconds or so, but I don’t want to just spam the channel with the updates; I want to compare the new update to the last one, and not post if the track hasn’t changed.

    I added a method to the metadata object to compare two objects:

    def __eq__(self, other) -> bool:
    if not isinstance(other, NowPlayingResponse):
        return False
    if other is None:
        return False
    return (self.dj == other.dj and
            self.artist == other.artist and
            self.track == other.track and
            self.album == other.album)

    …but I ran into a difficulty trying to store the old object: the async callback from my sseclient callback couldn’t see the variables in the main script. I knew I’d need a closure to put them in the function’s scope, and I was able to write that fairly easily after a little poking about, but even with them there, the inner function I was returning still couldn’t see the closed-over variables.

    The fix was something I’d never heard of before in Python: nonlocal.

    def wrapper(startup, last_response):
    def sender(response: NowPlayingResponse):
        nonlocal startup, last_response
        if response == last_response:
            return

        # Prepare the embed data
        local_tz = get_localzone()
        start = response.start.replace(tzinfo=local_tz)
        embed_data = {
            "title": f"{response.track}",
            "description": f"from _{response.album}_ by {response.artist} ({response.duration})",
            "timestamp": start,
            "thumbnail_url": response.artURL,
        }

        # Send to webhook
        send_webhook(embed_data)

        startup = False
        last_response = response

    return sender

    Normally, all I’d need to do would be have startup and last_response in the outer function’s argument list to have them visible to the inner function’s namespace, but I didn’t want them to just be visible: I wanted them to be mutable. Adding the nonlocal declaration of those variables does that. (If you want to learn more about nonlocal, this is a good tutorial.)

    The Discord monitor main code now looks like this:

    startup = True
last_response = None

# Build the SSE client
client = build_sse_client(server, shortcode)

# Create the sender function and start listening
send_embed_with_image = wrapper(startup, last_response)
run(client, send_embed_with_image)

    Now send_embed_with_image will successfully be able to check for changes and only send a new embed when there is one.

    One last notable thing here: Discord sets the timestamp of the embed relative to the timezone of the Discord user. If a timezone is supplied, then Discord does the necessary computations to figure out what the local time is for the supplied timestamp. If no zone info is there, then it assumes UTC, which can lead to funny-looking timesstamps. This code finds the timezone where the monitor code is running, and sets the timestamp to that.

    from tzlocal import get_localzone

local_tz = get_localzone()
start = response.start.replace(tzinfo=local_tz)

    And now we get nice-looking now-playing info in Discord:

    Shows two entries in a Discord channel, listing track title in bold, album name in italics, and artist name, with a start time timestamp and a thumbnail of the album cover.

    Building on this

    Now that we have a working Python monitor, we can now come up with a better solution to (close to) real-time updates for the iOS app.

    Instead of running the monitor itself, the app will register with the Python monitor for silent push updates. This lets us offload the CPU (and battery) intensive operations to the Python code, and only do something when the notification is pushed to the app.

    [Note: no, it’s not doing that.]

    But that’s code for next week; this week I need to get the iOS stopgap app out, and get the Python server dockerized.