Azuracast stream monitoring: the Greedy Shark

My ear is open like a greedy shark,
To catch the tunings of a voice divine.

• John Keats, Woman! when I behold thee flippant, vain

Why the Shark?

As one of the people managing RadioSpiral, I’m the one who’s in charge of the actual audio streaming server. One of our goals is to be up and running with an active audio stream 24/7. Most of the time, this is handled by an AutoDJ bot run by the Azuracast server.

We also have live DJs/performers who stream music to the station; when they connect, this pre-empts the AutoDJ.

Most of the time this all works well: the AutoDJ keeps tunes spinning, the DJs cut in to do their shows, and the AutoDJ takes over again when they disconnect.

Every once in a while, though, things don’t go as planned: network outages, DJ tech troubles, and the like. And when those happen, we go off the air.

Perhaps unsurprisingly, there are loads of tools for monitoring computer processes — is the server up, does it serve web pages, are there any errors — but almost nothing for monitoring audio.

So it was necessary to create one.

What the Shark does

The Shark exists to avoid the worst problem a radio station can have: dead air.

It’s a Python-based monitoring system that:

1. Actually listens to your stream using ffmpeg to capture audio samples and analyze them for silence
2. It knows the difference between “off the air” and “off the air because the DJ’s not sending audio” by using the AzuraCast APIs to check the server status and determine what’s going on.
3. It alerts our tech team via notifications in a private channel if the AutoDJ drops.
4. It takes action itself to force streamers off if they stop sending audio by suspending them via the AzuraCast API.
5. It has an associated Discord bot that lets the team check status and bring suspended streamers back without requiring a login to the AzuraCast server.

How It Works

To keep all this straight, I decided to use a state machine with three modes:

• NO_STREAMER (No DJ connected): after 2 minutes of silence send a “Silence detected!” message to a private Discord channel
• STREAMER_ACTIVE (DJ connected but silent): after 8 minutes of dead air, send a “suspension imminent” warning to the private channel, and after two more minutes, auto-suspend the streamer
• GRACE_PERIOD: (DJ is silent, but indicates they know it): monitoring paused for 15 minutes, preventing the DJ from getting booted while they’re sweating to fix their tech issue

This gives us some basic logic with room to expand it if we have new cases later.

Every 60 seconds, the Shark:

1. Captures a 10-second audio sample via FFmpeg
2. Analyzes RMS (volume) and variance (is it just silence or a stuck tone?)
3. Updates a consecutive silences counter
4. Checks whether the DJ has acknowledged a silence
5. Takes action based on current state

This gives us reasonable granularity and keeps the amount of data crunching down.

A paired Discord bot lets us query the state of the Shark, inform the monitor that we’re “working on it”, and unsuspend streamers who’ve been unable to or forgot to get streaming again.

• !shark-status – Current monitoring state and suspended users
• !working-on-it (or !woi)- Activate a 15-minute grace period
• !sharked – List all auto-suspended DJs
• !letin – Re-enable a suspended DJ
• !status – Check overall Shark status

The !woi command was the thing that turned the Shark from a somewhat annoying monitor into a useful DJ tool: if you get hung up, you know you can tell the Shark to leave you be for a bit while you fix stuff. And !letin keeps our less-technical DJs happy if they do happen to accidentally suspend themselves.

Open Source

The whole thing is on GitHub: https://github.com/joemcmahon/greedy-shark

MIT licensed, fully documented, with Docker Compose setup and comprehensive tests. Got an AzuraCast server? Try it yourself!

What’s Next

Other stuff we could do:

• Multi-stream support — we won’t need this, but someplace set up like SomaFM with a zillion different streams might want that. If we did that, a Web dashboard for monitoring history and status would be better than a Discord channel as the primary interface
• SMS alerts for critical issues — most of us leave our Discord alerts on enough that a ping from the Shark will get through, even late, and we’re sufficiently spread-out geographically that someone will see the alert, but serious “we are really, really down, like hours” alters should get sent in a “dude, you need to see this” way.

Try It Yourself!

If you run an internet radio station with AzuraCast, you can deploy this in about 10 minutes:

1. Clone the repo
2. Copy .env.example to .env
3. Fill in your AzuraCast API credentials and Discord target channel
4. Invite the bot to your server
5. docker compose up -d

And the Shark will be catching the tunings of your station.

Please let me know of any problems via the Issues on GitHub, have fun, and send patches if you do something interesting with it!