Extending DrQueue / DrQueueOnRails

General information

DrQueueOnRails is a new web frontend for DrQueue developed with RubyOnRails.

Current features:

  • Makes company/organization renderfarm setup with isolated jobs possible
    • User management (different status, permissions, disk quota, ...)
    • Quotas and different priority groups for users
    • LDAP authentication (optional)
    • Seperated working directories for each user and each job
    • Admin and demo account (for use without LDAP or testing)
  • The browser is the interface to the renderfarm
    • Upload of 3D scenes inside archive files (zip, tgz, tbz2, rar)
    • Preview of rendered frames if image type supported in browser
    • Download of finished images/animations
  • Also uses other Internet technologies
    • Info about status of jobs via email and RSS
  • Supported renderers: Maya (SW, MR), MentalRay standalone (scenes from Maya, 3DSMax, ImageStudio), Blender, Yafray, Cinema4D, V-Ray standalone (more renderers can be easily added)
  • Bigger files (HDRI, textures, ...) can be uploaded into a cache

Development

See a list of currently open tickets: https://ssl.drqueue.org/redmine/projects/drqueue/issues?query_id=1

Planned features:

  • Support for authentication against Active Directory (may include some LDAP issues)
  • Support upload of whole Maya project directory structures

Minimal requirements

  • some database server (tested with Sqlite3, MySQL and PostgreSQL)
  • Ruby 1.8.6+
  • Ruby bindings for DrQueue (see RubyBindingsHowto)
  • ImageMagick
  • some RubyGems: rails (version 2.3.5), ruby-net-ldap, rmagick, will_paginate, sqlite3-ruby (for Sqlite3) or postgres (for PostgreSQL) or mysql (for MySQL)
  • on the slaves you need to install the renderers you want to use

Recommended additional software

Try it out

You can find the source in the subdirectory 'DrQueueOnRails' in the Git repository at https://ssl.drqueue.org/git/drqueue.git .

Since version 0.64.4c2 of DrQueue, DQOR is also included in the source tarball.

Getting started

  • Note: this little guide assumes you are running a Unix-like system and are familiar with using a terminal/command shell
  • Add a user to your server who will run the webinterface:
    useradd -m -s /bin/bash drqueueonrails
    
  • Change to new user:
    su drqueueonrails -
    cd /home/drqueueonrails
    
  • Clone the Git repository:
    git clone https://ssl.drqueue.org/git/DrQueueOnRails.git DrQueueOnRails
    cd DrQueueOnRails 
    
  • Check for required software:
    ruby /home/drqueueonrails/DrQueueOnRails/DrQueueOnRails_requirements_check.rb
    
  • copy 'config/database.yml.example' to 'config/database.yml'
    cp config/database.yml.example config/database.yml
    
  • Optional: create a database and a database user and edit database.yml according to your database setup if not using Sqlite3
  • Copy 'config/environment.rb.example' to 'config/environment.rb' and edit it according to your DrQueue and LDAP setup
    cp config/environment.rb.example config/environment.rb
    
  • Change at least these variables:
    ENV['DRQUEUE_MASTER'] ||= 'MASTER'
    ENV['USER_ADMIN_PW'] ||= "changeme" 
    ENV['AVAIL_RENDERERS'] ||= "blender,blenderlux,cinema4d,luxrender,maya,mayamr,mentalray,vray" 
    
  • Recommended settings for testing/development:
    ENV['RAILS_ENV'] ||= 'development'
    ENV['WEB_PROTO'] ||= "http" 
    
  • Recommended settings for production:
    ENV['RAILS_ENV'] ||= 'production'
    ENV['WEB_PROTO'] ||= "https" 
    
  • Generate database tables
    rake db:migrate
    
  • Create the directories 'user_filecache' and 'dist_filecache' in DRQUEUE_TMP directory
    mkdir /usr/local/drqueue/tmp/user_filecache
    mkdir /usr/local/drqueue/tmp/dist_filecache
    
  • Optional: if you want to work with ImageStudio you should copy all material textures and HDR images from an ImageStudio installation to 'dist_filecache'
  • Optional: if you want to work with Maya and 3DStudioMax you should copy all their MentalRay include and library files into the directories of your standalone MentalRay installation
  • Optional: create a file 'public/contacts.html' and put some information in it to give your users a possibility to contact you
  • Optional: create a file 'public/disclaimer.html' and put some user policies in it
  • Start the web application
    script/server
    
  • Everything should be fine now and enter the URL to your server on Port 3000
  • Make sure you have cookies enabled
  • Login via 'demo' for account/password or 'admin'
  • If the login works, you should have a new entry in the 'profiles' table of your database
  • Note: You have to add your slaves for rendering for example with Blender to the pool 'blender'. This also applies to all other supported renderers.

Using LDAP for authentication

  • LDAP can provide authentication for a lot of services
  • Most bigger companies/organizations with lots of users run LDAP already, so DQOR can integrate here
  • Edit environment.rb:
    ENV['LDAP_TREEBASE'] ||= "dc=drqueue, dc=org" 
    ENV['LDAP_HOST'] ||= "SERVER" 
    ENV['LDAP_PORT'] ||= "389" 
    ENV['LDAP_FILTER'] ||= "uid" 
    ENV['LDAP_ATTRS'] ||= "mail,cn" 
    
  • Set 'LDAP_TREEBASE' to the identifier of your LDAP
  • Set 'LDAP_HOST' to the hostname/ip address of your LDAP server
  • Set 'LDAP_FILTER' to the attribute which is used to distinguish users from each other
  • Restart the web application and try to login with a LDAP account
  • Set your status attribute in the SQL database (not LDAP) to 'admin' to have administrative rights and relogin

Using Apache+Passenger instead of Webrick/Mongrel

  • Apache is often run on webservers and well known (a complex but mighty monster ;) )
  • Passenger is a module which integrates RubyOnRails applications nice into Apache
  • Install Passnger and mod_xsendfile Apache modules
  • Add a virtual host entry to your Apache config:
    <VirtualHost *:80>
      ServerName renderfarm
    
      TransferLog /home/drqueueonrails/AccessLog
      ErrorLog /home/drqueueonrails/ErrorLog
      DocumentRoot /home/drqueueonrails/drqueue.git/DrQueueOnRails/public
    
      <Directory /home/drqueueonrails/drqueue.git/DrQueueOnRails/public>
        Options +FollowSymLinks
        Options +SymLinksIfOwnerMatch
        AllowOverride All
      </Directory>
    
      XSendFile on
      XSendFileAllowAbove on
    
      RailsEnv development
    </VirtualHost>
    

Things to consider for bigger setups

  • use LDAPS instead of plain LDAP
  • use HTTPS instead of plain HTTP
  • think about cluster filesystems for advanced performance/caching

Screenshots

  • Note: These screenshots are a little bit older.

screenshot_01.jpg - login page (39.6 KB) Andreas Schröder, 10/29/2007 02:18 pm

screenshot_02.jpg - after login (82.9 KB) Andreas Schröder, 10/29/2007 02:21 pm

screenshot_03.jpg - list of jobs (81.8 KB) Andreas Schröder, 10/29/2007 02:21 pm

screenshot_04.jpg - job details (61.8 KB) Andreas Schröder, 10/29/2007 02:24 pm

screenshot_05.jpg - new single image job (46.5 KB) Andreas Schröder, 10/29/2007 02:25 pm

screenshot_06.jpg - new animation job (48.3 KB) Andreas Schröder, 10/29/2007 02:26 pm

screenshot_07.jpg - profile details (45.1 KB) Andreas Schröder, 10/29/2007 02:28 pm

screenshot_08.jpg - list of computers (87.1 KB) Andreas Schröder, 10/29/2007 02:29 pm

screenshot_09.jpg - admin only: list of all jobs (110 KB) Andreas Schröder, 10/29/2007 02:30 pm

screenshot_10.jpg - admin only: list of all jobs (second part) (123 KB) Andreas Schröder, 10/29/2007 02:31 pm

screenshot_11.jpg - admin only: list of all profiles (55.5 KB) Andreas Schröder, 10/29/2007 02:32 pm

screenshot_12.jpg - admin only: edit profile (46.9 KB) Andreas Schröder, 10/29/2007 02:33 pm

screenshot_13.jpg - admin only: computer details (52.6 KB) Andreas Schröder, 10/29/2007 02:34 pm

screenshot_14.jpg - advanced users & admins only: manage filecache (72 KB) Andreas Schröder, 10/29/2007 03:12 pm