Mythbackend deadlocks and auto-restart script

I recently bought two new DVB-C cards (two Technotrend C-1500). Until then I was using Anysee E30 Combo Plus. So now I have three cards altogether. As always, Linux and DVB hardware is a whole story in itself, I’ll write more about this upgrade in a later post. But anyway, I eventually got all three cards working together.

However, there seems to be some deadlock issue in mythbackend when you use multiple DVB cards. I’ve previously used multiple tuners for one card and never had an issue with that configuration. After I upgraded to three cards, every now and then (weekly) when a recording is ongoing and a new one is about to start or has just started, MythTV will deadlock. What I mean by deadlock is that it becomes completely unresponsive from mythfrontend and mythweb. Logs stop. There’s no particular error in the logs that would indicate any kind of problem. I do get an entry for the recording in the database, but the file it is pointing to doesn’t exist.

I found a ticket that sounds suspiciously similar, but the issue was fixed in revision 22814. According to the release notes, that fix should be in MythTV 0.23, which I’m using. Also, since the recording entry is inserted into the database, one might assume that it is not a scheduling issue?

I’m hoping this issue will go away with future upgrades, like these issues usually do. To handle the situation for now, I made a script to automatically restart mythbackend if it deadlocks or dies. This is a good idea to have even if your setup is relatively stable. You don’t want mythbackend to die shortly after you leave for that two-week vacation, and thereby miss all recordings, do you?

This is the mythbackend deadlock restart script I made:

#!/bin/sh
# print status
STATUS=`/sbin/service mythbackend status | grep -E "mythbackend .*is running"`
echo `date`: $STATUS
FILEDATE=`date +%Y%m%d_%H%M%S`
mv -f /tmp/mythbackend-status /tmp/mythbackend-status.previous
RESULT=`wget --timeout=10 --output-document=/tmp/mythbackend-status --retry-connrefused --tries=4 http://127.0.0.1:6544 2>&1`
if [ "$?" -ne "0" ]
then
        mv /tmp/mythbackend-status.previous /tmp/mythbackend-status.crashed.$FILEDATE
        echo -- ERROR: mythbackend is not responding... must restart: $RESULT
        echo `/sbin/service mythbackend status`
        echo `/sbin/service mythbackend stop`
        echo `/sbin/service mythbackend start`
        echo `date`: Restart done
else
        echo Service is responding: $RESULT | sed "s/Saving.*//"
        rm /tmp/mythbackend-status.previous
fi

You might need to change some mythbackend commands in there, so that they work for your setup. Now just run the script as a root cronjob every minute. I have made the script so that it takes less than one minute to execute, otherwise you can’t have a cronjob that runs every minute. Remember this if you want to modify the script. Here’s the crontab:

* * * * * * /root/checkmythbackend.sh >> /tmp/mythbackend.log

So, if something happens, your mythbackend will be up and recording again within a minute. The script will generate the following files in your /tmp folder:

-rw-r--r-- 1 root root 1337325 2011-01-10 23:28 /tmp/mythbackend.log
-rw-r--r-- 1 root root   11700 2011-01-10 23:28 /tmp/mythbackend-status
-rw-r--r-- 1 root root    8963 2011-01-09 18:30 /tmp/mythbackend-status.crashed.20110109_183101

The first one is that log for the restart script. The second file is a temporary file containing the latest state of mythbackend. The third one contains the state of mythbackend before the crash/deadlock. There you can see all ongoing and upcoming recordings, and all jobs etc.

P.S. I have a similar ‘deadlock restart script’ for my current digital receiver (Nokia 260C). It is in the form of an electricity timer that cuts the power for one minute every morning :) I have had a lot of stability issues with that one, probably because it was one of the first recording digital receivers on the market.