Here we will focus on the most current developments for KGHP following a painful outage in November 2015. We've experienced a similar outage before, and will surely experience an outage of similar magnitude again. Thanks to new technology implementations, future outages will have a smaller impact on the station's legality and listener satisfaction.

Why?

Well, KGHP serves a much greater purpose than simply letting students play on the radio. There are hundreds of community members who tune in for various reasons. Underwriters (Sponsors) pay good money to have their messages heard and the station needs to abide by FCC guidelines. FCC fines are much greater than the man hours it takes to add multiple layers of redundancy. We are in a place now where proper monitoring is in place and actually have multiple instantaneous failovers tested. Let's take a look at the system as a whole.

Status surrounding outage

The outage in November was inevitable. The Airtime servers hadn't been power cycled since they were put in place, and their drives were significantly degraded. The primary server that had been pumping music for over a year had two failed drives and no chance of data recovery. Software backups to the secondary server had stopped working sometime in May-June and even throughout the Fall months when I felt the need to check the backup, I didn't find time. Due to insufficient monitoring and checking, we had a backup, but it was sour and full of corrupt files. After the first few hours after the outage it was apparent I wasn't going to get Airtime online with the current status of the backup. It had a failed drive and the OS running on the RAID was causing significant issues.

We had to run illegally...

After several ours of dead air and inability to start the AutoDJ, our only solution was to queue up a few thousand songs on the vault in a loop playlist. KGHP has contractual obligations to play underwriter messages. It also has to meet FCC guidelines by announcing the station ID at the top of every hour. Obviously the simple playlist couldn't do either of these. It took around 6 weeks to get Airtime playing again, and that was only part of the battle. We quickly realized there were still large amounts of corrupt files.

New implementation

KGHP Diagram made with an iPad Pro and Apple Pencil Above is a sketch of the critical KGHP technologies that Tech Services is responsible for as of this writing.

AutoDJ starts at Airtime. We have a redundant set of Overpowered Dell Work Stations. Each has a 4 2TB disk RAID and a boot drive connected directly to the motherboard. The boot drive prevents OS corruption if the RAID fails or degrades, and provides consistency for backups and restoring.

KGHP Monitored Services

Notice the monitored RAID status on the Airtime Hosts. This is done by a megacli script that uses drivers for communicating RAID status from the Dell RAID Controllers. Any warning or failure this should be a priority for drive replacement. These are warranted by Dell beyond 2025 (extended warranty purchased).

Airtime creates an audio stream to the network. This stream is listened to by the low energy Raspberry Pi. The Raspberry Pi acts as a load balancer. It plays Airtime A stream out of it's connected soundcard at all times. If Airtime A is quiet, it streams from Airtime B. If Airtime B is quiet it starts playing music from it's SD card.

KGHP Raspberry Pi

As long as the Raspberry Pi has 5v of power, it will play music indefinitely and LEGAL for KGHP. I experimented with implementing backup power using a 1s6p 18650 battery back from a "powerbank" USB interface and added a 3rd party charger for always on availability. However, the charger I used wasn't strong enough, at around 500mah charging (Advertised 1000mah). The raspberry Pi and soundcard consumed about 1,500 to 2,000mah. The charger wasn't able to keep the Lipo cells topped off while supporting the load. A larger 2sXp battery at 7.5v with 5v step down would be needed to supply the load and keep cells topped off. As charging a 1s pack over 3,000mah would be nonstandard and likely dangerous.

How it streams

Liquidsoap is an amazing tool for programming audio. Using the below Liquidsoap script, the Raspberry Pi acts like an intelligent audio load balancer. Outputting the resulting stream to it's USB soundcard.

Located at /home/pi/radio.liq

#!/usr/bin/liquidsoap

# STL Liquidsoap Config
# Logging

set("log.file.path", "/home/pi/log/liquidsoap.log")  
set("decoding.buffer_length", 30.)

# Setup our sources

#primary airtime server
main_url = "http://airtimea.kghp.org:8000/airtime_128"  
#backup airtime server
backup_url = "http://airtimeb.kghp.org:8000/airtime_128"

emergency_audio = playlist(reload=3600,"/home/pi/emergency/music.m3u")

underwritters_audio = playlist(reload=10800, "/home/pi/emergency/urw.m3u")

top_audio = playlist(reload=10800,"/home/pi/emergency/top.m3u")

#Copy music playlist
radio_emg = emergency_audio

#Add underwritter jingles
radio_emg = rotate(weights = [1, 5],[underwritters_audio, radio_emg])

#Add top of the hour jingles
radio_emg = add([radio_emg,switch([({0m0s},top_audio)])])

#Error detection
def silenceHandler()  
system("echo 'should log'");  
end

# Our main source is streaming with silence detection

radio_stream = input.http(id="PrimaryStream", main_url)  
radio_stream = on_blank(silenceHandler, radio_stream)  
radio_stream = strip_blank(max_blank=10.0,threshold=-30.0,radio_stream)

backup_radio_stream = input.http(id="BackupStream", buffer=10.0, max=80.0, backup_url)  
backup_radio_stream = strip_blank(max_blank=10.0, threshold=-30.0,backup_radio_stream)

# Fall back to an MP3 file

#Output airtime A, fallover to airtime B, fallover to generated playlist
radio_stream = fallback(track_sensitive = false, [radio_stream, backup_radio_stream, radio_emg])

# Finally, pump it out the soundcard
output.alsa(fallible = true, radio_stream)  

In the future the Raspberry Pi could take incoming sound from the studio, then send the stream to mainstream for online streaming including tune-in. Currently this is being done by a Windows 2000 machine :( This however would require the USB soundcard for audio in. The raspberry Pi's 3.5mm is PWM digital, and not analog (Same reason there's no analog GPIO pins).

Mainstream (and it's monthly cost) would be eliminated by utilizing purely the PSD nginx load balancers for https stream URLS: https://psd401.net/stream.