This is the first time KGHP has had this level of redundancy. In the past there wasn't enough time or knowledge to successfully mirror the two Airtime systems to both play at the same time. The only issue up to now was failover, which required boots in Spencer's office to swap the USB cable. However thanks to a Raspberry Pi, we've added a whole new level of redundancy.

The idea is simple. Instead of having the soundcard plugged into one of the airtime servers, have it instead plugged into a small reliable device (Like one of the three Raspberry Pi's we have here) where the sound output is simply a stream from the currently available airtime servers. This would give us more flexibility in location as well, we could move the primary Airtime server to our server room and have the backup isolated at PHS. A small low energy device like the Raspberry Pi could be run off of a battery backup for hours, automatically switch streams if one is down with virtually no dead air, and could even have a local backup playlist on the SD card play if neither one of the airtime streams are available. With the sound output device monitored with Nagios, points of failures are significantly minimized and we gain certainty in always being legal.

Already experienced hardware issues

After weeks of configuring Airtime and ensuring the music library was clean, we already had soundcard issues after only two weeks.

Sunday January 10th I spent about 2 hours on the phone with Spencer after we had an hour of dead air. I did what I could remotely, but was unable to get the soundcard responding again. It appeared as if we had some kind of power outage over the weekend. A switch at maintenance failed too. The primary Airtime server was failing with the same errors that I was dealing with in December. The error before was ultimately resolved by wiping and re-installing the OS. I am still unsure why this was happening, I've asked the communities for help, but the solution presented to me was to split the hardware output from the Airtime server.

Both systems were outputting to icecast (creating a digital stream) just fine, but when the hardware output option was checked, Airtime came to a halt. When I asked Airtime support about the issue, they told me that they never run a production Airtime system directly to hardware. Instead using icecast to stream to another client to output to the station board.

The backup system wasn't responding to the usb soundcard either. If I had a good backup of the OS disks for Airtime, I feel the issue could have been resolved easily, but unfortunately I hadn't gotten that far. After spending two hours over the weekend, and 2 more hours Monday morning fighting with it, I put a Raspberry Pi to good use.

Configuration

I spent about 3 Hours that morning configuring the Raspberry Pi, then testing when the Dell tech was at KGHP. I quickly found a perfect liquidsoap script that served our purpose extremely well and modified it to Spencer's requirements.

WARNING:

Liquidsoap and liquidsoap-plugin-all need to be installed before running the script. Note, this takes a LONG time on a First generation Raspberry Pi.
sudo apt-get install liquidsoap liquidsoap-plugin-all

See /home/pi/radio.liq

#!/usr/bin/liquidsoap

# STL Liquidsoap Config
# Logging

set("log.file.path", "/home/pi/log/liquidsoap.log")

# Setup our sources

#primary airtime server
main_url = "http://airtimea.kghp.org:8000/airtime_128"  
#backup airtime server
backup_url = "http://airtimeb.kghp.org:8000/airtime_128"

emergency_audio = playlist(reload=3600,"/home/pi/emergency/music.m3u")

underwritters_audio = playlist(reload=10800, "/home/pi/emergency/urw.m3u")

top_audio = playlist(reload=10800,"/home/pi/emergency/top.m3u")

#Copy music playlist
radio_emg = emergency_audio

#Add underwritter jingles
radio_emg = rotate(weights = [1, 5],[underwritters_audio, radio_emg])

#Add top of the hour jingles
radio_emg = add([radio_emg,switch([({0m0s},top_audio)])])

#Error detection
def silenceHandler()  
system("echo 'should log'");  
#Could do something else here, just not sure what as Nagios is already monitoring the stream for silence
end

# Our main source is streaming with silence detection

radio_stream = input.http(main_url)  
radio_stream = on_blank(silenceHandler, radio_stream)  
radio_stream = strip_blank(max_blank=5.0,threshold=-30.0,radio_stream)

backup_radio_stream = input.http(backup_url)  
backup_radio_stream = strip_blank(max_blank=5.0, threshold=-30.0,backup_radio_stream)

# Fall back to an MP3 file
# Also worth noting, if we get this far, we ought to tell someone (if we can)

#Output airtime A, fallover to airtime B, fallover to generated playlist
radio_stream = fallback(track_sensitive = false, [radio_stream, backup_radio_stream, radio_emg])

# Finally, pump it out the soundcard
output.alsa(fallible = true, radio_stream)  

In around 50 lines including comments, the script connects to Airtime and streams the output to a soundcard. If Airtime A is silent or offline, it fails over to Airtime B. If Airtime B is silent or offline, it fails over to a local playlist.

Notice the strip_blank function, this was added after failover was only happening when the stream was unavailable. Basically what this command does is listen to the stream's decibel amount and if the output is less than -30DB for 5 seconds, failover is initiated. For example, if Airtime is unscheduled or has a problematic file, but is still technically online, the Raspberry Pi will catch this as an error and failover to it's local playlist to prevent dead air.

The local playlist plays 5 songs, then an underwritter via the rotate function. Then adds a random top of the hour jingle at every hour. The only issue we saw with this, is the top of the hour is not queued in the playlist, instead played on top of whatever is playing at that moment. This is slightly disorientating. A solution undoubtedly exists, but is not a priority as we don't plan on relying on this system except for emergencies. However in case of emergency, at least we have a dumb little autoDJ that can take over.

See these links for scripting references:
http://savonet.sourceforge.net/doc-svn/quick_start.html
http://savonet.sourceforge.net/doc-svn/cookbook.html
http://hackaday.com/2014/04/17/raspberry-pi-remote-audio-link/
http://www.dlineradio.co.uk/articles/stls-and-raspberry-pis/

A cronjob keeps three local folders in sync with the data backup: underwriters, top of the hour, and an emergency music folder. the --delete toggle ensures music removed from these folders also get removed from the SD Card.
See /home/pi/create_backup.sh

#!/bin/bash

rsync -rav --delete /backups/labeled/urw/ /home/pi/emergency/urw/  
rsync -rav --delete /backups/labeled/top/ /home/pi/emergency/top/  
rsync -rav --delete /backups/labeled/emergency/ /home/pi/emergency/music/

find /home/pi/emergency/music/ -type f -name "*.mp3" > /home/pi/emergency/music.m3u  
find /home/pi/emergency/urw/ -type f -name "*.mp3" > /home/pi/emergency/urw.m3u  
find /home/pi/emergency/top/ -type f -name "*.mp3" > /home/pi/emergency/top.m3u  

Where /backups/ is a mount from //10.0.70.24/radiobacks

The last 3 lines generate the playlist files for liquidsoap. Basically a list of available mp3 files.

I tested the system and it works great except the minor top of the hour bug. This was a small time investment for the value it adds. We now have a fully redundant system where I will have backups of the raspberry pi SD card too. If for whatever reason, we have Airtime outages, the station will remain legal as long as that pi has a 5v input.