Richard BF

There are probably a dozen different things that can go wrong with Apple’s Compressor and Qmaster, and the web is filled with solutions for most of them. Except mine that is. So today I ripped apart Qmaster to find out once and for all what the problem is.

For reference sake, Qmaster is Apple’s queuing and distributed processing technology. Compressor, Apple’s multipurpose AV transcoder which comes with Final Cut and DVD Studio, uses Qmaster to queue and distribute audio and video transcoding tasks across multiple Macs in a network. Sounds pretty advanced doesn’t it? I mean a home business or hobbyist is probably not going to want to run a server farm dispatching service, when all they want to do is to convert their new video to MPEG2 (for DVD) or some iPod format. Problem is, that’s the default install for both Final Cut Pro and Final Cut Studio.

The problem with Qmaster, is when you export a video sequence from Final Cut to Compressor. Compressor asks which format you’d like to transcode to, and then tries to call Qmaster to make it all happen. Typically you then get an error dialog which says “Unable to connect to background process”, at which point Compressor cancels the job, and Final Cut returns you to editing mode.

Some solutions to this problem suggest exporting from Final Cut to an intermediary format, and then using Compressor stand alone to transcode, but this doesn’t bypass the problem. The problem isn’t with Final Cut or Compressor, it is with Qmaster.

Qmaster is required by Compressor, not Final Cut. One of the things that takes a while when debugging Qmaster, is running Final Cut everytime, creating a sequence, exporting to Compressor, setting the transcoding format, and then finally clicking submit. There’s an easier way to see if the problem has been fixed.

Launch Compressor on its on, don’t bother with Final Cut. Go to Compressor’s Preferences, and change Default Destination to Desktop, change Cluster Options to Never copy source to Cluster, and set the Default Setting to Web Download (QuickTime 7 compatible) –> H.264 100Kbps, then click Apply. Exit from Compressor, and drag a small .jpg picture to your desktop. Now you’re ready to go.

To test if the “Unable to connect to background process” problem has been fixed, launch Compressor on its own, select Import file… from the File menu, click on the .jpg on the desktop, and click Open. This will return you to the Compressor Batch window, where you can simply click on Submit. If you don’t see the error dialog, then you’re done, and please leave a comment here to say what worked for you.

If you still have a problem, use the above test after each thing you test from the following huge list of possible solutions.

For the rest of this post, I was originally using Final Cut Studio 5.1.1, but have since gone back to 5.0 with the 5.0.4 update.

Qmaster the dark art

Qmaster starts up when your Mac starts up. In fact there’s a startup shell script in /Library/StartupItems/Qmaster on your boot volume, called Qmaster, which at Mac boot time grabs the file /Library/Application Support/Apple Qmaster/qmasterconfig, which contains the pathname of the Qmaster daemon (/usr/sbin/qmasterd), and some basic configuration variables. It thens launches the daemon.

At this point, the Qmaster daemon should be running. Once the boot has finished, you can launch Terminal and enter top to see the process list, which should include a number of related Qmaster processes, including one or more instances of qmasterd. An easier way to check if Qmaster is running, is to use finder to launch /Applications/Utilities/Activity Monitor, select All Processes from the select menu, and then enter qmaster into the search field. You should at least get one instance of qmasterd.

If Qmaster is not running, then there’s a few things you can try to get it running.

Qmaster is not running

1. Apple recommends the first thing you do is reinstall everything, but sometimes this can make things worse if you don’t delete everything, as you can end up with mixed versions of base install and subsequent versions downloaded through Software Update. If you’re going to reinstall, then reinstall everything the way Apple recommends, but I would only use this as a last resort.
2. Often a reinstall may have excluded Qmaster. To check, go to Terminal and type:
   

   ls -l /Library/Application Support/Apple Qmaster/

   Note that there are space characters between “Application” and “Support”, and “Apple” and “Qmaster”, and the slash “” character is used to escape them. You could instead wrap the entire path in double quotes if you wish. Now, if you don’t see a list of Qmaster related files, such as qmasterconfig or QmasterApplications.plist, then you haven’t installed Qmaster, and you’ll need to go back to your base install disks, and reinstall.

3. It could be possible that a reinstall didn’t actually add Qmaster to the startup items folder. To check this, use finder to open /Library/StartupItems, and make sure the Qmaster folder exists, and contains the startup script. You should also check the security permissions. See below.
4. Sometimes the security permissions for Qmaster’s startup script get munged. To fix this, go to Terminal, and type the following:
   1. cd /Library/StartupItems/Qmaster
   2. rm -f .disabled
   3. sudo chmod 755 Qmaster
   4. sudo chown root:wheel Qmaster
   5. sudo chmod 644 StartupParameters.plist
   6. sudo chown root:wheel StartupParameters.plist

   This also removes a possible .disabled file, which can also disable Qmaster from starting. You may also need to run /System/Library/CoreServices/SecurityFixer to fix permissions, however I’ve seen very little feedback that this helps at all. You will need to reboot your Mac to actually start Qmaster, and test if it is now working.

5. Another problem with starting Qmaster, is sometimes the software it requires is not available, such as AECore services and the Qmaster.framework. Go back to the install disks and make sure you’ve installed everything correctly. Alternatively, you can use Pacifist (a third party standalone package installer) to only install the Qmaster package. Different versions of Final Cut will package things differently, so you need to be careful which packages you need to install. If in doubt, use Pacifist to install the lot.
6. Qmaster requires three main system services to operate, Disks, Resolver (DNS) and Network. These are listed in /Library/StartupItems/StartupParameters.plist. I doubt the startup items would be called without these, but perhaps third party software is causing these not to start before Qmaster is started. If this is the case, either start up Qmaster manually (see below), or find the offending software and remove it or get a newer version.

If Qmaster still won’t start, then try starting it manually. Go to Terminal and type:

cd /Library/StartupItems/Qmaster

Then type:

sudo /Library/StartupItems/Qmaster/Qmaster start

You’ll get a message like Starting Qmaster services, which just means that the script has tried to run. It may take up to a minute to start, but eventually it should appear in top or the Activity Monitor. If this works for you, then you’ll need to restart it again whenever you reboot.

If it still won’t start, then there’s probably some software which is missing, and I’d recommend reinstalling everything. Although there is an error log for Qmaster at /Library/Logs/Qmaster/qmasterd.log, if any of the lines within it start with Error:, then again it is probably some software which is missing.

One additional thing to try, is removing all network connections. If you’re not using a server rendering farm, just the one computer you edit on, then Qmaster won’t require network access. While there are several theories as to why this fix works, I’ve yet to find a consistent and reputable one, but it generally goes like this: when Qmaster starts up, it looks for network connections on one or more known ports, and if it can’t find them, it fails to start up. This doesn’t make much sense technically, and the solution to the problem is just as convoluted, but it seems to work for some people. It goes like this: if you shut down all network services, then Qmaster won’t go looking for ports, and it will startup without failing.

There are three ways you can do this, that I found on the web. Two you can try, and a third you should not.

1. Go to the System Preferences, Network pane, select Network Port Configurations from the Show: select menu, and uncheck all the Port Configurations, then click on Apply Now. This will disable all your network services, and you can then in theory try to restart Qmaster. You can either reboot and see if it starts automatically, or use the manual method of starting it with sudo from Terminal as described above. Again, check with top or Activity Monitor to see if it starts.
2. Instead of killing all your network connections, go to System Preferences, Network pane, and create a new Location: setting from the Location: select menu. Call it “Qmaster” or “Compressor” or “[no network]”, whatever makes sense to you, and then again uncheck all the Port Configurations in Network Port Configurations. You now have your standard Location: for normal networking, and your new special Qmaster settings for when you need Qmaster. Of course whenever you have the Qmaster one active, you have no network capability, so no Internet access.
3. Some web forums have suggested that you remove /etc/hostconfig. DO NOT DO THIS! This file is required by your Mac when you start up, and removing it will cause your Mac to permanently boot into the system shell. It should fix the Qmaster problem, because it disables all networking, but it also disables pretty much everything else as well, and with no GUI, you’re going to find it hard to use Compressor anyway. Some have also suggested copying this file to the desktop, renaming it to hostconfig.old and then copying it back to /etc, which does nothing. Some people have reported success in playing with hostconfig, however my thought is that they did something else at the same time which fixed it, or it was a permission problem which was fixed by touching hostconfig. My recommendation is to not try it, although if you’re interested you can find numerous descriptions of it in Google.

Qmaster is running

At this point, you should be confident that Qmaster is actually running, so the problem may well be mixed versions of Compressor and Qmaster. If you think this is the case, then follow Apple’s reinstall method above, and make sure your versions are in sync.

If this is not the problem, then it could be that the Qmaster configuration is wrong. This is the problem that I was having. I’ve never changed the Qmaster preferences, but for some reason, something changed them, maybe an install of another Apple product, maybe a crashed or cancelled Compressor session, who knows.

Final Cut Studio apparently includes a System Preferences pane for Qmaster. I saw it once, but when I clicked on it, it crashed, so I’ve never actually seen it. Subsequent installs of Final Cut Studio haven’t installed it again. Final Cut Pro doesn’t include this, so you won’t see it. These preferences are used to configure the network which Qmaster will use for distributed rendering. If you’re only using your own Mac, or your preferences are corrupted, then this configuration may be wrong, causing Qmaster to fail when trying to distribute jobs.

Luckily, there is a shell version of Qmaster preferences, which comes with all versions. To see your current configuration, go to Terminal and type:

cd /usr/sbin/

… to get to the right directory. Then type:

qmasterprefs -list

… to see your current configuration.

You should see settings for one or more clusters, plus two services, Compressor Processing and Rendering. While qmasterprefs is documented in both the Qmaster and Compressor manuals, the service parameters are not, and neither are most of the cluster parameters. I’m assuming this is because users typically go through the GUI, which most people may not actually have installed.

First thing is to make a copy of the current settings, so you can restore them, so do a qmasterprefs -list, copy the output into a text editor, and save it to disk for later.

Now, we need to make sure we have a single machine defined. My machine name is g5, so I have a cluster started on g5 as follows:

-cluster on servername "g5" quickcluster on unmanagedservices on unmanagedmulticapturethreshold 0 networkinterface allinterfaces

If your machine name is wrong, then change it with:

qmasterprefs -cluster on servername "yourmachinename"

Your machine name will be listed at the beginning of the prompt in Terminal. e.g. my prompt looks like this:

[g5:/usr/sbin] richardbf%

g5 is my machine name, /usr/sbin is the directory I’m currently in, and richardbf is my user name.

You may also need to enable quick clustering using:

qmaster -cluster on quickcluster on

Next, you need to make sure Qmaster is enabled for Compressor and Rendering services. To do this, type:

qmasterprefs -service "Compressor Processing" on instances 1

… to enable Compressor, and then:

qmasterprefs -service "Rendering" on instances 1

… to enable rendering. Note that my initial prefs had these two turned off, and an instances of -1 for Rendering. As these are not documented, I don’t know the exact settings they should be, but these work for me.

The networkinterface allinterfaces may also need to be set for your cluster, in case that has been changed as well.

Here’s the configuration that works for me:

[g5:/usr/sbin] richardbf% qmasterprefs -list
Current settings:
-cluster on servername "g5" quickcluster on unmanagedservices on unmanagedmulticapturethreshold 0 networkinterface allinterfaces
-service "Compressor Processing" on instances 1 autorestart off unmanaged on
-service "Rendering" on instances 1 autorestart off unmanaged on
[g5:/usr/sbin] richardbf%

You can also type qmasterprefs -help for help on which commands you can use.

Once you’ve made your changes, double check that they’d been accepted, by reissuing:

qmaster -list

Now restart Qmaster by typing:

qmasterprefs -restart

… then go back and try Compressor to see if the error dialog still appears.

One thing to note, Compressor automatically detects your single cluster, and populates the Cluster: select menu in the Batch window, with your machine name. That’s the select menu next to the Submit button. When Compressor starts, this will be set to None, then as it connects to Qmaster, it should quickly change to your machine name. If not, then you should select it manually yourself. If your machine name isn’t in the list, then go back and make sure you set your -cluster options correctly.

Either way, you should also reboot your Mac and try Compressor again, just to be sure. If it works before the reboot but not after, then you may have a security permissions problem, or perhaps the installation is still flakey, in which case I’d suggest reinstalling everything as mentioned above.

If it doesn’t work before a reboot, but does after, or to be honest who cares so long as it works after, then you’ve fixed the problem.

Conclusion

I cannot believe that this is still a problem. Surely Apple would know of all the problems people are having, both in their support forums and elsewhere on the web, and would redesign the installation and configuration of both Qmaster and Compressor.

Subsequent versions of Final Cut Studio and Compressor do include newer versions of Qmaster, however in my experience, an upgrade makes it very easy to end up back in a whole world of “Unable to connect to background process” pain. My recommendation is, if you’ve got a running fairly up to date version of Final Cut and Compressor working, then don’t upgrade unless you absolutely need features or fixes in the newer version, and even then, only upgrade to minor versions, not a major one.

iPod support

Version 2.11 of Compressor includes a new built in setting for transcoding videos for the iPod, and gives you options such as using either H.264 or MPEG4 as the encoding format. This is a great setting to have, but in my case I’ve had to forego it in the name of Qmaster woes. Instead, I have to use the Convert to iPod setting in iTunes to convert videos to iPod format. Unfortunately this always uses MPEG4 and not H.264, but at least it works.

If you have any additional comments or solutions, then please let me know. My system is now working, but it took a lot of effort to try everything, and then come up with my own, seemingly new solution.