Home > General > Mythbackend deadlocks and auto-restart script

Mythbackend deadlocks and auto-restart script

January 11, 2011 Leave a comment Go to comments

I recently bought two new DVB-C cards (two Technotrend C-1500). Until then I was using Anysee E30 Combo Plus. So now I have three cards altogether. As always, Linux and DVB hardware is a whole story in itself, I’ll write more about this upgrade in a later post. But anyway, I eventually got all three cards working together.

However, there seems to be some deadlock issue in mythbackend when you use multiple DVB cards. I’ve previously used multiple tuners for one card and never had an issue with that configuration. After I upgraded to three cards, every now and then (weekly) when a recording is ongoing and a new one is about to start or has just started, MythTV will deadlock. What I mean by deadlock is that it becomes completely unresponsive from mythfrontend and mythweb. Logs stop. There’s no particular error in the logs that would indicate any kind of problem. I do get an entry for the recording in the database, but the file it is pointing to doesn’t exist.

I found a ticket that sounds suspiciously similar, but the issue was fixed in revision 22814. According to the release notes, that fix should be in MythTV 0.23, which I’m using. Also, since the recording entry is inserted into the database, one might assume that it is not a scheduling issue?

I’m hoping this issue will go away with future upgrades, like these issues usually do. To handle the situation for now, I made a script to automatically restart mythbackend if it deadlocks or dies. This is a good idea to have even if your setup is relatively stable. You don’t want mythbackend to die shortly after you leave for that two-week vacation, and thereby miss all recordings, do you?

This is the mythbackend deadlock restart script I made:

#!/bin/sh
# print status
STATUS=`/sbin/service mythbackend status | grep -E "mythbackend .*is running"`
echo `date`: $STATUS
FILEDATE=`date +%Y%m%d_%H%M%S`
mv -f /tmp/mythbackend-status /tmp/mythbackend-status.previous
RESULT=`wget --timeout=10 --output-document=/tmp/mythbackend-status --retry-connrefused --tries=4 http://127.0.0.1:6544 2>&1`
if [ "$?" -ne "0" ]
then
        mv /tmp/mythbackend-status.previous /tmp/mythbackend-status.crashed.$FILEDATE
        echo -- ERROR: mythbackend is not responding... must restart: $RESULT
        echo `/sbin/service mythbackend status`
        echo `/sbin/service mythbackend stop`
        echo `/sbin/service mythbackend start`
        echo `date`: Restart done
else
        echo Service is responding: $RESULT | sed "s/Saving.*//"
        rm /tmp/mythbackend-status.previous
fi

You might need to change some mythbackend commands in there, so that they work for your setup. Now just run the script as a root cronjob every minute. I have made the script so that it takes less than one minute to execute, otherwise you can’t have a cronjob that runs every minute. Remember this if you want to modify the script. Here’s the crontab:

* * * * * * /root/checkmythbackend.sh >> /tmp/mythbackend.log

So, if something happens, your mythbackend will be up and recording again within a minute. The script will generate the following files in your /tmp folder:

-rw-r--r-- 1 root root 1337325 2011-01-10 23:28 /tmp/mythbackend.log
-rw-r--r-- 1 root root   11700 2011-01-10 23:28 /tmp/mythbackend-status
-rw-r--r-- 1 root root    8963 2011-01-09 18:30 /tmp/mythbackend-status.crashed.20110109_183101

The first one is that log for the restart script. The second file is a temporary file containing the latest state of mythbackend. The third one contains the state of mythbackend before the crash/deadlock. There you can see all ongoing and upcoming recordings, and all jobs etc.

P.S. I have a similar ‘deadlock restart script’ for my current digital receiver (Nokia 260C). It is in the form of an electricity timer that cuts the power for one minute every morning :) I have had a lot of stability issues with that one, probably because it was one of the first recording digital receivers on the market.

Advertisements
  1. Larry
    May 24, 2011 at 22:22

    I’m very happy about that script. I’ll had deadlocks also with 0.24.1 but not as usually as with older two versions (and older that 0.23 I have never used). It seems that the more I have recording entries and saved recording the more there is deadlocks. Sometimes it seems that there is also database deadlocks. Recording is not always stopped if there is deadlock, but that situation is not very usual.

    Perhaps more powerful computer is better, I feel that multiple cores and/or HT would be better than old single core and single thread system when database, backend and frontend are all on the same machine.

    Your blog is now in my favorites, its nice to see that somebody shares things about this software tweaks.

  2. Larry
    May 30, 2011 at 22:54

    I made some modifications to script and cronjob. Just an 40 minutes ago cronjob was restarting mythbackend every one minute. It needs few minutes to get all up and running.

    At the start of the script add this:
    # check if mythbackend is running, we don’t wan’t to start process that has shutdown by admin
    if ! [ “$(pidof mythbackend)” ]
    then
    exit
    fi

    Change restart to look like this:
    echo `/usr/sbin/service mythbackend status`
    echo `/usr/sbin/service mythbackend stop`
    /bin/sleep 5
    echo `/usr/sbin/service mythbackend start`
    echo `date`: Restart done
    /bin/sleep 120

    Perhaps also the waiting time on wget should be set longer on some systems.

    And cronjob should not be run if script is already running:
    * * * * * /usr/bin/flock -n /var/lock/checkmythbackend.lck /usr/local/bin/checkmythbackend.sh >> /var/log/mythtv/checkmythbackend.log

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s