Documentation / Setup DrQueue / CentOS setup guide

Install the base system

  1. Boot from your first CentOS 5.4: CD (CD 1) or the CentOS 5.4: DVD. Press <ENTER> at the boot prompt.
  2. It can take a long time to test the installation media so we skip this test by hitting tab and enter. The welcome screen of the CentOS installer appears. Click on Next.
  3. Choose your language next.
    english
  4. Select your keyboard layout.
    US
  5. I'm installing CentOS 5.4: on a fresh system, so I answer Yes to the question Would you like to initialize this drive, erasing ALL DATA?
    Now we must select a partitioning scheme for our installation. For simplicity's sake I select Remove linux partitions on selected drives and create default layout. This will result in a small /boot and a large / partition as well as a swap partition. Then I hit Next. Answer the following question (Are you sure you want to do this?) with Yes.
  6. On to the network settings. The default setting here is to configure the network interfaces with DHCP, but we are installing a master, so static IP addresses are important... Click on the Edit button at the top right.
  7. In the window that pops up uncheck Use dynamic IP configuration (DHCP) and Enable IPv6 support and give your network card a static IP address (in this tutorial I'm using the IP address 192.1:68.1:.100 for demonstration purposes) and a suitable netmask (e.g. 255.2:55.2:55.0; if you are not sure about the right values, http.//www.subnetmask.info might help you).
  8. Set the hostname manually, e.g. Master, and enter a gateway (e.g. 192.1:68.1:.1:) and up to two DNS servers (e.g. 68.1:80.1:31.1:6 and 68.1:42.2:55.1:6). From this point this will appear as [SERVER NAME] in later text.
  9. Choose your time zone.
    Chicago
  10. Give root a password. something alpha numeric

Configure our setup

  1. On the install menu, don't check Packages from CentOS Extras. Then check Customize now, and click on Next.
  2. Now we must select the package groups we want to install. Select Gnome Desktop, Editors, Text-based Internet, Development Libraries, Development Tools, MySQL Database, Server Configuration Tools, Web Server, Base, and System Tools (unselect all other package groups) and click on Next.
  3. When the installer finishes checking, click on Next to start the installation.
  4. The hard drive will be formatted, and the installation begins. This will take a few minutes.
  5. When the installation is complete, and you can remove your CD or DVD from the computer and reboot it.
  6. After the reboot, you will see a Setup agent screen. Select Firewall configuration and disable it.
  7. SELinux is a security extension of CentOS that should provide extended security. You don't need it, so disable it. Hit yes when prompted.
  8. Create one new user account for drqueueonrails.
  9. Click forward on sound. it wont really matter much.
  10. Leave the Setup Agent by selecting Exit.
  11. Your system will now reboot. Login as drqueueonrails. Click on system and go to Preferances when you get to a desktop.
  12. Go to remote desktop. Make sure there is a check for sharing to allow remote desktop and allow users to control the desktop.
  13. Now go to System at the top menu. Then go to administration and finally login. On the users tab. Put a check in include all users from /etc/passwd. On the remote tab set it to same as local. Then go to security and enable auto login. Set drqueueonrails as the login. Reboot the system so that your changes can be applied.
  14. Connect with SSH to [SERVER] using a terminal on another station:
    ssh 192.1:68.1:.XXX (XXX will be the IP you set the server to) 
    

Create users

  1. Using a terminal on another station, shell into the master system. Use the username and the password you created in step 1.8:
    ssh drqueueonrails@192.168.1.100
    [PASSWORD]
    su
    [PASSWORD]
    
  2. Create user accounts for anyone that needs access.
    adduser -m [YOU]
  3. Create passwords for the user accounts you just created.
    passwd [YOU]
    
  4. Make sure users have admin access. You can do this by editing the /etc/group file, and adding users to the wheel group. Save your changes by hitting control+x.
    nano /etc/group
    wheel:x:10:root,drqueueonrails,[username]
    

Adjust hosts file

  1. We edit /etc/hosts. Make it look like this. Type vi /etc/hosts to begin editing. REMEMBER [SERVER NAME] is a specific name, like farmer101 or master.
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1:  localhost.localdomain localhost [SERVER NAME].example.com [SERVER NAME]
    192.1:68.1:.1:00  [SERVER NAME].example.com [SERVER NAME]
    ::1  localhost6.localdomain6 localhost6
    

Install some software

  1. First we import the GPG keys for software packages, so we type:
    rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY*
    
  2. You need to install an additional RPM
    rpm -Uhv http://apt.sw.be/redhat/el5/en/i386/RPMS.dag/rpmforge-release-0.3:.6:-1.el5.rf.i386.rpm
    rpm -Uhv http://apt.sw.be/redhat/el5/en/i386/RPMS.dag/rpmforge-release-0.5.1-1.el5.rf.i386.rpm
    
  3. We need to create an additional repo file for some additional software the farm will need.
    nano /etc/yum.repos.d/epel.repo
    
  4. Now that we have a file open, we paste the new repo location and save it.
    [epel]
    name=Extra Packages for Enterprise Linux 5 - $basearch
    #baseurl=http://download.fedoraproject.org/pub/epel/5/$basearch
    mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-5&arch=$basearch
    failovermethod=priority
    enabled=1
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
    
  5. And we grab the RPM key
    cd /etc/pki/rpm-gpg
    wget http://fr.rpmfind.net/linux/EPEL/RPM-GPG-KEY-EPEL 
    
  6. Update our existing packages on the system:
    yum -y update
    
  7. Now we install some software packages that are needed later on. While you are in the shell paste the following command to your terminal:
    yum -y install wget bzip2 unzip zip nmap openssl lynx ncftp gcc gcc-c++ httpd httpd-devel mysql-server pkgconfig php php-devel php-gd php-imap php-ldap php-xml php-xmlrpc curl curl-devel perl-libwww-perl libxml2 libxml2-devel mod_python ntp gtk2-devel gtk2 python-devel swig scons tcsh pygtk2-devel pkgconfig vnc-server fuse fuse-ntfs-3g gnome-session ImageMagick mod_xsendfile blender
    
    reboot
    

Create shares and resources

  1. We are going to start by building the share. You need this to put the DrQueue software someplace reachable. To do that we need to create a folder. Switch over to super user and we can get started.
    mkdir /mnt/shared
    
  2. Once thats done we can mount the drive to its share, later. Lets move on to some other fun stuff.

Enable packet forwarding on the master

  1. Login as the root user. Open /etc/sysctl.conf file
    vi /etc/sysctl.conf
    
  2. Add the following line to enable packet forwarding for IPv4:
    net.ipv4.conf.default.forwarding=1
    
  3. Save and close the file. Restart networking:
    service network restart
    
  4. To share network connection via eth1, enter the following rule at command prompt (following useful for ppp0 or dial up connection):
    service iptables stop
    iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
    service iptables save
    service iptables restart
    

Ruby On Rails install

  1. The first thing we need to do is add the ruby to the repository. to do that we edit the yum.repos.d.
    nano /etc/yum.repos.d/rubyworks.repo
    
  2. Insert this text into your new rubyworks.repo.
    # Link file for RubyWorks yum repository
    # Place this file in /etc/yum.repos.d
    
    [rubyworks]
    name=RubyWorks
    baseurl=http://rubyworks.rubyforge.org/redhat/$releasever/RPMS/$basearch
    enabled=1
    gpgcheck=1
    gpgkey=http://rubyworks.rubyforge.org/RubyWorks.GPG.key
    priority=1
    
  3. Now we need to install the ruby components that will run the rails. So we type the following command.
    yum -y install ruby ruby-devel ruby-libs ruby-irb ruby-rdoc ruby-mysql
    
  4. RubyOnRails has to have a few more things. Like rails. To install rails we need to install gems. RubyGems is the equivilant of apt-get or yum for RubyOnRails. We install it like so.
    cd /usr/src
    wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
    
  5. Once downloaded you need to open it up to see whats inside.
    tar -zxf rubygems-1.3.5.tgz
    
  6. Now lets get this thing installed and compiled. The process takes a while to complete so you need to be patient with this.
    cd rubygems-1.3.5
    ruby setup.rb
    
  7. Alright. Now you are going to need some source files for your rubygems. Lets tell ruby where they are.
    gem sources -a http://gemcutter.org
    gem sources -a http://gems.rubyforge.org
    gem sources -a http://gems.github.com
    gem update --system
    
  8. Wow, that was time consuming. But we arent done. Now we can install our required components. These take time too. Go get some coffee.
    gem install rails 
    gem install ruby-net-ldap
    gem install rmagick 
    gem install mislav-will_paginate
    gem install mongrel
    

MySQL setup

  1. To set a password for the user root (otherwise anybody can access your MySQL database!). Run the following:
    is what you set as a password.
    WRITE IT DOWN. Few things are as annoying as looking for a lost MySql Password.
    mysqladmin -u root password [yourrootsqlpassword]
    mysqladmin -h server1.example.com -u root password [yourrootsqlpassword]
    
  2. We need databases in MySql.
    mysql -u root -p 
    

    #You will be prompted for your password that you just entered. After you put that in you will get the 'Welcome to MySql' banner. Now you can make a database. Type the following:
    CREATE DATABASE renderdata; 
    CREATE USER 'drqueueonrails'@'localhost' IDENTIFIED BY PASSWORD '[userpassword]';
    
  3. This is your database. Write it down!. You are going to need this alot. Now you have to create a user that going to use the database. So we type the following and create a user and give access at one time. Remember '[]' is variable data
    GRANT ALL ON *.* TO '[USER]'@'localhost' IDENTIFIED BY '[PASSWORD]';
    

Startup links

  1. We create the system startup links for Apache (so that Apache starts automatically whenever the system boots) and start the Apache server:
    /sbin/chkconfig --levels 235 httpd on        
    /etc/init.d/httpd start
    
  2. Then we create the system startup links for MySQL (so that MySQL starts automatically whenever the system boots) and start the MySQL server:
    /sbin/chkconfig --levels 235 mysqld on
    /etc/init.d/mysqld start
    

Create Additional Resources

  1. Alright then, you have your master system. Its going to be the showcase of your work. It is going to be the 'Face-Man'. Now we need to get the 'lackeys' that are going to be the real workhorses created. To this end we are goingh to repeat Section Steps 1.00: through 7.00: for every system that we intend to use as a slave. Point of note. You will need one system that is going to act as that shared resource we mentioned in Section 7.00:. So, go do that and get back to me when you are done. One more thing. Besure to set the individual hostname for each render node, ie: farmer101, farmer102, etc.
  1. So the slaves are created. We can mount the drive to its share. This is going to be an instrumental part of what we are doing. Just about everything we do is going to do is going to be from this shared resource. Because I have twenty-one systems I am going to use the last one as a share.
    ssh [username]@192.1:68.1.121
    [password for [username]]
    su
    [password for root]
    mkdir /mnt/shared 
    
  2. There will be an export file needed to make your shares work. Go to /etc, and you will need to nano the exports file. Once it it open you need to add every machine that will need access to this share. Like so,
    nano /etc/export
    
  3. List each machine you wish to have connect to the share.
    /mnt/shared/    192.168.1.100(rw) 192.168.1.101(rw) 192.168.1.102(rw) etc...
    
  4. OR you can cover a whole subnet with;
    /mnt/shared 192.168.1.0/255.255.255.0(rw)
    
  5. Either of these will make it so that your systems will be able to connect to the shared drive/resource when it is available.
  6. After we have a matching directory on our Master, the slaves and our resource system. So now we need to go back to our master. You can ssh to the entire farm network if you did it right. Actually this is a good test of your system. You will need to network to every machine from the master to do this next step. We are going to mount the shared resource. As root you need to mount the share.
    mount 192.168.1.121:/mnt/shared /mnt/shared
    
  7. In order to make this permanant you need to edit the fstab and add the following.
    192.168.1.121:/mnt/shared /mnt/shared nfs rw,hard,intr 0 0
    

Setup master and slaves

  1. Now we need to tell the machines what they are going to do. We have to set the enviromentals and the startups for the systems to run the farm. To that end we need to edit the bashrc on all of the machines. While logged in as root we are going to do an edit. Once in the bashrc you need to past in your enviromentals. This will tell each machine who's incharge and where find him.
    PATH=${PATH}:/mnt/shared/drqueue/bin
    export PATH
    
    export DRQUEUE_ROOT=/mnt/shared/drqueue
    export DRQUEUE_MASTER=192.168.1.100
    
  2. We will also need to edit the rc.local also. This is going to tell every machine to start the farm engines automatically. To do the we need to get into the /etc/rc.local. Once you nano into it, paste in your parameters. Oh, and dont restart yet. We need to wait a bit.
    PATH=${PATH}:/mnt/shared/drqueue/bin
    export PATH
    
    export DRQUEUE_ROOT=/mnt/shared/drqueue
    export DRQUEUE_MASTER=192.168.1.100
    
    /mnt/shared/drqueue/bin/slave
    
  3. The rc.local for the master is different. This is going to tell every machine to start the master engine automatically. To do the we need to get into the /etc/rc.local. Once you nano into it, paste in your parameters. Again dont restart it yet.
     
    PATH=${PATH}:/mnt/shared/drqueue/bin
    export PATH
    
    export DRQUEUE_ROOT=/mnt/shared/drqueue
    export DRQUEUE_MASTER=192.168.1.100
    
    /mnt/shared/drqueue/bin/master &
    

DrQueue setup and install

  1. Mmmmm.. Looks good so far. Now lets get to our main point. Install the DrQueue. You need to shell out to the resource system. In our example its 192.68.1.121
            
    ssh [USER]@192.168.1.121
    [PASSWORD]
    su
    [PASSWORD]
    cd /usr/src
    wget http://www.drqueue.org/files/drqueue.0.64.3.tgz
    tar -xzf drqueue.0.64.3.tgz
    cd /drqueue-0.64.3
    scons PREFIX=/mnt/share install
    cd python
    python setup.py install
    
  2. Drqueue will have to have Master and slave configs that tell the renderfarm where to find things. firste we edit the master.conf, then the slave and finally the drqman.conf.
    #Master Config
    logs=/mnt/shared/drqueue/logs
    tmp=/mnt/shared/drqueue/tmp
    db=/mnt/shared/drqueue/db
    bin=/mnt/shared/drqueue/bin
    etc=/mnt/shared/drqueue/etc        
    
    #Slave Config 
    logs=/mnt/shared/drqueue/logs
    tmp=/mnt/shared/drqueue/tmp
    pool=[POOLNAME]
    pool=Default:Linux
    #logs=./logs
    #tmp=./tmp
    
    #Drqman Config
    logs=/mnt/shared/drqueue/logs
    tmp=/mnt/shared/drqueue/tmp
    db=/mnt/shared/drqueue/db
    

Restarts and Ruby bindings

  1. DrQueue is installed. Now we need to install the ruby bindings. This will be a little complicated. Hang on tight and we can get through this. Switch over to the drqueueonrails user.
    su drqueueonrails
    [PASSWORD]
    gem update --system
    cd /home/drqueueonrails
    gem install --install-dir /home/drqueueonrails DrQueueRubyBindings
    
  2. We will need to setup some environment variables and some database varilbles also. So we nano our database.yml first, and set our points.
    nano /home/drqueueonrails/gems/DrQueueRubyBindings-0.2/ext/0.64.x/DrQueueOnRails/config/database.yml 
    
    development:
    adapter: mysql
    database: renderdata
    username: drqueueonrails
    password: [userpassword]
    
  3. Then we save that and edit our environment.rb to fit our needs. NOTE: I am disabling LDAP and Https because I dont use it. You are free to use it if you want but, I cant help you there.
    nano  /home/drqueueonrails/gems/DrQueueRubyBindings-0.2/ext/0.64.x/DrQueueOnRails/config/environment.rb
    # DrQueue variables
    ENV['DRQUEUE_MASTER'] ||= '192.168.1.150'
    ENV['DRQUEUE_TMP'] ||= '/mnt/shared/drqueue/tmp'
    ENV['DRQUEUE_ETC'] ||= '/mnt/shared/drqueue/etc'
    ENV['DRQUEUE_LOGS'] ||= '/mnt/shared/drqueue/logs'
    
    # LDAP variables
    #ENV['LDAP_TREEBASE'] ||= "dc=mydomain, dc=de" 
    #ENV['LDAP_HOST'] ||= "SERVER" 
    #ENV['LDAP_PORT'] ||= "389" 
    #ENV['LDAP_FILTER'] ||= "uid" 
    #ENV['LDAP_ATTRS'] ||= "mail,cn" 
    
    # protocol for webserver (HTTP, HTTPS)
    ENV['WEB_PROTO'] ||= "http" 
    
  4. Setup the webpage to see the DrQueue interface in your httpd.conf. This way you can add jobs to the farm. You will need to nano your httpd.con and do some editing.
    nano /etc/httpd/conf/httpd.conf
    

    #Paste in the parameters for the website. Use something like this.
    <VirtualHost *:80>
        ServerAdmin webmaster@dummy-host.example.com
        DocumentRoot /home/drqueueonrails/gems/DrQueueRubyBindings-0.2/ext/0.64.x/DrQueueOnRails/public/
        ErrorLog logs/drqueueonrails-error_log
        CustomLog logs/drqueueonrails-access_log common
    
        <Directory /home/drqueueonrails/gems/DrQueueRubyBindings-0.2/ext/0.64.x/DrQueueOnRails/public/>
          Options +FollowSymLinks
          Options +SymLinksIfOwnerMatch
          AllowOverride All
        </Directory>
    
        RewriteEngine On
    
        ProxyPass /images !
        ProxyPass /stylesheets !
        ProxyPass /player !
        ProxyPass /javascripts !
        ProxyPass /upload !
    
        Alias /images /home/drqueueonrails/gems/DrQueueRubyBindings-0.2/ext/0.64.x/DrQueueOnRails/public/images
        Alias /stylesheets /home/drqueueonrails/gems/DrQueueRubyBindings-0.2/ext/0.64.x/DrQueueOnRails/public/stylesheets
        Alias /player /home/drqueueonrails/gems/DrQueueRubyBindings-0.2/ext/0.64.x/DrQueueOnRails/public/player
        Alias /javascripts /home/drqueueonrails/gems/DrQueueRubyBindings-0.2/ext/0.64.x/DrQueueOnRails/public/javascripts
        Alias /upload /home/drqueueonrails/gems/DrQueueRubyBindings-0.2/ext/0.64.x/DrQueueOnRails/public/upload
    
        ProxyPass / http://127.0.0.1:3000/
        ProxyPassReverse / http://127.0.0.1:3000/
        ProxyPreserveHost on
        ProxyRequests Off
    
        RequestHeader set X_FORWARDED_PROTO 'http'
    </VirtualHost>
    
  5. Almost there. In order for the DrQueue on rails to run it has to be started. So we need to edit our rc.local. Add this to the bottom of it. This way everytime the system reboots it will auto-start the DrQueue web interface.
    export DRQUEUE_ETC=/etc/drqueue
    export DRQUEUE_TMP=/mnt/share/drqueue/tmp
    export DRQUEUE_MASTER=192.168.1.100
    cd /home/drqueueonrails/gems/DrQueueRubyBindings-0.2/ext/0.64.x/
    mongrel_rails start -d
    

Modify DrQueueOnRails

  1. We have to make a small modification. The webface is going to need to be able to find a file so lets make it easy. make sure you do this as root.
    cp /home/drqueueonrails/gems/DrQueueRubyBindings-0.2/ext/0.64.x/etc/blender_sg.py /mnt/render/drqueue/etc/
    
  2. Also a little thing I found is that the web interface needs something extra. It need access to the tmp folder.
    chmod -R drqueueonrails:drqueueonrails tmp/
    

Look at your work

  1. Now that you have reached this stage we are ready for the final part. You are going to have to shell to every machine in your farm from the master and restart them. Yes they have power buttons you can reach, but you need to do it this way. Shelling into each box from the master ensures you will know if something/someone is down. So shell to every system and restart them. Then restart your master.
  2. Go to a different machine and open a VNC-Viewer. You are going to connect to the master with it and see whats going on. Make sure you know the IP of the master. The digit after the colon is a desktop number. Since we are using a default login we are setting ourselves to desktop '0'. Like this.
    vncviewer 192.168.1.100:0
    
  3. Once you log into the desktop of the master load up a browser. Go to http://localhost:3000. You should get a DrQueue Web Interface. When you get the interface login with demo:demo. Try it out. Look around at your work. You did it.