Please note: This was the initial reboot in 2014. This configuration lasted around year before hardware failures exposed the short comings. Some of this is still valid, but not everything.
See the updated redundancy post here.

Key focal points

  • No more Gnome GUI, use Debian minimal install terminal only
  • Use vmnocdbpostgr02 for database
    • Has Daily backups in Veam
    • 1Gig fiber connection to server room
  • Server A and B /data directories automatically rsynced over the network for music backup and redundancy
  • Server A and B both running airtime with rsynced configurations and pointing to the same remote database

Process

Lost all data... Need to start from scratch on both boxes.

Setup OS

  • Setup Raid on both machines with new Raid Controllers
  • Partition the virtual disk, 100gb for OS, 8gb for Swap, rest for mounting to the /data directory
  • Re-install the minimal OS to the primary boot partition
  • Set static IP and forward DNS entry airtime.kghp.org to IP
  • Add /etc/fstab config to mount /data directory
  • setup 5 minute reoccuring cron job to backup and sync /data directory accross computer A and B
  • Setup email alerts for RAID health
  • Setup init.d startup scripts for auto starting services and maintinance
  • Setup RAID monitoring
      -

Install Airtime

  • Install Airtime with apt-get
  • Setup username and password same as before
  • Point airtime database to vmnocpostgr01
    • username: airtime
    • password: kghp89.9
    • databasename: airtime
    • dumped database from default airtime installation and restored on postgres server to airtime database
  • Test external audio card with default icecast/alsa configuration.
  • Set music and watch directory to /data/labeled
  • Set Music import directory to /data/music

Manage Music

  • Transfer all music from all Spencer's portable hard drives to /data/vault
  • Setup SAMBA share for access to music library over SMB
    • Mountable on Spencer's computer
    • Usable by Mixxx in the future
  • re-create automated music uploading
    • Setup ssh keys
    • Installed eyeD3 for metadata editing
    • Scripts are still on Spencer's computer for using scp to copy, convert and label music.
      • Connects to server and pulls all current labels
      • Displays labels in a multiselect applescript dialog
      • Uses scp to move the song into the label's folder
      • Uses eyeD3 to set the metadata label on the MP3 (Airtime currently does not import this field)
  • Recreate script and setup on cron job to keep labels synced with postgres database.
    • Setup on 5 minute cron job
    • Written in PHP
    • connects to vmnocdbpostgr02 database
    • queries for cc_files that do not have labels
    • Takes the file location and parses the label from the folder tree
    • Does an UPDATE to the postgres record

How To Recover from a server failure

If you find yourself in the position of losing all the data or receiving dead air from the primary server, here are the steps you should take.

  1. Unplug the gray usb cable from the primary server and plug it into the backup server.
  2. reboot the secondary server

After the secondary reboots, you should be back on the air playing the current scheduled show

Creating a new server from scratch

  1. Grab the latest debian installer and burn the iso to a flash drive
  2. Proceed to install it on the server.
  3. Set root password to TS admin password
  4. Partition the disk:
    • 100gb for OS
      • Mount point /
      • ext3 format
    • 8gb for swap
    • Rest of disk for data
      • Mount point /data
      • ext3 format
  5. When prompted to install software, deselect debian desktop. Only have ssh server and Standard system utilities checked.
  6. After the installation is done, login as root
  7. Add deb http://apt.sourcefabric.org/ lucid main to the /etc/apt/sources.list then apt-get update
  8. Install airtime and dependencies apt-get install airtime
    • Use dedicated V-host (Should look into a central web server...)
    • Set servername to it's hostname
    • "no change" to apache config
    • Say yes to icecast automated passwords
    • Leave icecast hostname as localhost
    • Set all icecast passwords
    • Airtime storage directory: /data/music
    • Set Airtime Admin password
    • Yes configure icecast
    • Set all icecast passwords again
  9. Install Alsa apt-get install alsa
  10. comment out line: options snd-usb-audio index=-2 to allow the usb card to be loaded as default
  11. edit the /etc/airtime/airtime.conf file and change the [database] config
    • host = vmnocdbpostgr02
    • dbpass = kghp89.9
  12. Add ServerAlias to /etc/apache2/sites-available/
  13. reboot