Extending DrQueue / Python bindings Howto

Building the Python bindings

Basic usage

You have examples of the use of the python bindings in directory python/examples and python/DrKeewee of the sources.

To browse those examples online:

Object types (data structures for libdrqueue)

The two most important object types are the job and the computer. They represent a job (either already in the queue or not yet sent) and a computer in the queue, that is a registered slave.

As in Python everything is an object, we take advantage of that. The original C library deals with C structures. But thanks to SWIG, we can extend those data structures with methods to treat them like objects. As mentioned, they're objects in Python.

Job object

A job object stores all job related information like the job status, frames left and done, number of active processors... as well as the limits related to that job like the maximum number of cpus, the pool to which this job is assigned, etc. It also has methods to perform actions with it.

We can obtain job instances either requesting the list of jobs to the master or creating a job instance ourselves.

Example: Requesting the list of jobs to the master

>>> # Version >= 0.64.2
>>> import drqueue.base.libdrqueue as drqueue
>>> job_list = drqueue.request_job_list(drqueue.CLIENT)
>>> len (job_list)


>>> # Version < 0.64.2
>>> import drqueue
>>> job_list = drqueue.request_job_list(drqueue.CLIENT)
>>> len (job_list)

As you can see the result is a python list. The environment variables DRQUEUE_MASTER and DRQUEUE_MASTER should be set, same way as for any other drqueue command line tool.

Example: creating a job instance

import drqueue
# OR if version >= 0.64.2
#import drqueue.base.libdrqueue as drqueue

job = drqueue.job()

Then we can modify it, adjusting values to be sent.

Python 2.4.1 (#2, Jul 21 2005, 15:58:35)
[GCC 4.0.2 20050720 (prerelease) (Debian 4.0.1-2ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import drqueue
>>> job = drqueue.job()
>>> job.name = "Test python job" 
>>> job.frame_start = 1
>>> job.frame_end = 10
>>> job.cmd = "ls -al" 
>>> job.priority = 100
>>> job.limits.pool = "foobar" 
>>> job.send_to_queue()
1    <----- indicates success
>>> job.id
4    <----- the job id number assigned by the master

The are some exceptions implemented to deal with errors. But do not rely only on them yet, try to check the return codes.

Job Methods

(The argument who will be drqueue.CLIENT)

  • job.request_frame_list(who) : Returns a python list of frame_info object instances
  • job.job_frame_index_to_number (index) : Returns the frame number asociated to a frame_info index
  • job.request_continue (who) : Returns 1 on success
  • job.request_stop (who) : Returns 1 on success
  • job.request_hard_stop (who) : Returns 1 on success
  • job.request_delete (who) : Returns 1 on success
  • job.request_rerun (who) : Returns 1 on success
  • job.send_to_queue () : Returns 1 on success
  • job.update (who) : Will contact the master to update the information about the status of the job. (Not including the frame list). Returns 1 on success.

the Computer object

The computer object stores all information related to the computer (including limits, pools, tasks...) as well as methods to perform actions on them.

As a difference with jobs, there is no use (atm) in creating a computer instance directly. They're only useful when you receive them from the master with drqueue.request_computer_list(drqueue.CLIENT)


>>> import drqueue
>>> computer_list = drqueue.request_computer_list(drqueue.CLIENT)        <---------- retrieves the list of slaves from the master
>>> len (computer_list)                                                  <---------- as a python list
>>> computer = computer_listr0                                          <---------- we get the first instance
>>> computer.hwinfo.name                                                 
'localhost'                    <---------- it's name
>>> computer.hwinfo.ncpus
1                              <---------- number of cpus
>>> computer.limits.enabled
0                              <---------- is the slave enabled for rendering ? nopes, it's disabled. 
>>> computer.request_enable(drqueue.CLIENT)                              <---------- let's enable it then.
>>> computer.update(drqueue.CLIENT)                                      <---------- we sent the request but, did it work ? the update method will retrieve the information from the master, so we can check.
>>> computer.limits.enabled                                              <---------- is enabled it now ?
1                              <---------- yes, that computer will be active now. :)

In this example we can see how to retrieve some basic information about one of the slaves connected to the master. It's name, number of cpus and enabled status.

With the method request_enable we request the slave to change it's enabled status to 1 (enabled). With method update as well as with the method of the same name in the job, we update the information stored in the object computer from the master.

Note: When we request the slave to change it's status to enabled, we are doing so directly to the slave, NOT to the master. That means that if we write a program that changes the status of the slaves and right away requests the information to the master, it won't probably be updated. It takes some seconds to the slave to report it's status to the master.

computer methods
  • computer.request_enable(who)
  • computer.request_disable(who)
  • computer.update(who)

Other useful info

Independently of the object's method that are python specific we have access to all libdrqueue functions. So we can call any function using same type of arguments the C function call would use.

For example:

    drlist = drqueue.request_computer_list(drqueue.CLIENT)        # retrieves the list from the master. raises exception on error.
    drlist = None
    print "Could not retrieve list of slaves" 

found = 0
for tcomp in drlist:
    if tcomp.hwinfo.name == 'slave001':     # compare it's name as a string
        compid = tcomp.hwinfo.id            # get the id as an integer
        found = 1

if found:
    # this function requests the slave named 'slave001' to set it's limit of available cpus to 4
    rcode = drqueue.request_slave_limits_nmaxcpus_set ('slave0001',4,drqueue.CLIENT)

In this sample code you can see how a regular library function is called: drqueue.request_slave_limits_nmaxcpus_set (<string>,<integer>,drqueue.CLIENT)

Any other function can be used the same way.

Be careful though with double pointer arguments. Those will probably fail or corrupt your data. I have written code to deal with some, but it's not complete.