Using EC2 with DrQueue

Aim

DrQueue provides a solid renderfarm control environment, and Amazon Web Services EC2 provides nodes on demand. It makes sense that the two services should work together and function in such a manner that AWS EC2 can provide an on demand render farm for projects.

David's work so far

As it stands at the moment, some basic scripts have are being developed in order to deploy DrQueue in to a EC2 instance and get it up and running. These will be made more public in the near future when a suitable location has been found for them. There is still a manual level of work required which it is felt should be automated in order to reduce any issues with the deployment.

Discussions about this development can be found on http://randomtechblogs.wordpress.com (Along with other things).

It should be noted that the scripts are currently going through a redesign phase, so the scripts found on the blogs can be used but at the users own risk.

Long term aim

The long term aim will be to have some software which acts between the user and the cloud environment, providing an automated process of connecting DrQueue slave VMs to each other. Job submission should work with little or no intervention. Slaves should be started on demand.

Planned Features:
  • Renderfarm administrators can setup rendersessions by user's request (choose type of VM, number of rendering slaves, hours of rendering time).
  • Users can add jobs and run them while their rendersession is valid.
  • All data is stored on encrypted storage volumes.
  • All communication between master and slaves is done encrypted accross a VPN.
  • Users can have their own storage volumes.
Usable VM types:
  • Large (L) Instance (m1.large)
    • 7.5 GB RAM
    • 4 EC2 Compute Units
    • 850 GB instance storage (2 x 420 GB plus 10 GB root partition)
    • 64 Bit platform
    • I/O performance: high
  • Extra Large (XL) Instance (m1.xlarge)
    • 15 GB RAM
    • 8 EC2 Compute Units
    • 1.690 GB instance storage (4 x 420 GB plus 10 GB root partition)
    • 64 Bit platform
    • I/O performance: high
  • High-CPU Extra Large (XL) Instance (c1.xlarge)
    • 7 GB RAM
    • 20 EC2 Compute Units
    • 1.690 GB instance storage (4 x 420 GB plus 10 GB root partition)
    • 64 Bit platform
    • I/O performance: high

DrQueue on EC2 - status

  • DONE:
    • A testbed for development is currently at Amazon is provided by kaazoo (contact me if interested!)
    • AMI is working for master and slaves, but needs to be enhanced here and there
    • Master: OpenVPN server, DrQueue, DrQueueOnRails, DrQueueCloudControl, VPN key generation runs
    • Slave: provisioning script, OpenVPN client, CIFS client, DrQueue slave runs
    • rendering test with 20 slaves rendering a simple Blender animation
    • Backup: add to BackupPC or find another Backup solution
    • Authentication: LDAP server for authentication
    • Connectivity: Apache / Nginx with Passenger, HTTPS
    • Connectivity: add public IP address and DNS entry
    • File system security: create user directory below DRQUEUE_TMP and mount user directory there
  • TODO for master:
    • Stability: enhance functionality and stability of DrQueueCloudControl
    • File system security: disk encryption of EBS storage device
    • Storage: single EBS volumes for extra separation of single users
  • TODO for slaves:
    • Stability: test and enhance mass-provisioning
    • Versatility: add more free renderers

Reference implementation

BlenderCloud is a public rendering service and based on this howto. See http://www.blendercloud.net for more information.

DrQueue on EC2 - setup howto

Base OS

  • Create t1.micro instance from public AMI 'ami-10794c64' (Ubuntu 10.04 LTS 64bit, released 2010-09-23)
  • Login via key pair as user ubuntu

Dependencies

  • Install Ubuntu packages:
    aptitude install ruby rdoc ri ruby-dev git-core swig scons pkg-config libsqlite3-ruby 
                     libopenssl-ruby libmagick++-dev build-essential openvpn samba smbfs nginx
                     unzip mysql-server libmysqlclient-dev
    
  • Install RubyGems-1.3.7 from tarball
  • Install RubyGems:
    gem install net-ldap rails(2.3.8) DrQueueRubyBindings will_paginate rmagick amazon-ec2 mysql
    

Secure storage

  • smbpasswd -a drqueue

Secure communication

  • Setup Easy-RSA on master:
    cd /etc/openvpn/
    cp -av /usr/share/doc/openvpn/examples/easy-rsa .
    cd easy-rsa/2.0
    nano vars
    source ./vars
    ./clean-all
    ./build-dh
    ./pkitool --initca
    ./pkitool --server myserver
    
  • Setup OpenVPN server on master:
    cd /etc/openvpn/
    cp /usr/share/doc/openvpn/examples/sample-config-files/office.up server.up
    cp /usr/share/doc/openvpn/examples/sample-config-files/tls-office.conf server.conf
    cp easy-rsa/2.0/keys/myserver.crt .
    cp easy-rsa/2.0/keys/myserver.key .
    cp easy-rsa/2.0/keys/dh1024.pem .
    cp easy-rsa/2.0/keys/ca.crt .
    nano server.conf
    /etc/init.d/openvpn restart
    
  • Keystore:
    mkdir /var/www/keystore
    

DrQueue configuration

  • Setup special user and install software:
    useradd -m -s /bin/bash drqueue
    su drqueue -
    cd
    git clone https://ssl.drqueue.org/git/drqueue.git
    cd drqueue
    scons build_drqman=false
    exit
    scons install
    
  • run DrQueue master:
    su drqueue -
    /usr/local/drqueue/bin/master.Linux.x86_64
    
  • run DrQueue slave:
    su drqueue -
    /usr/local/drqueue/bin/slave.Linux.x86_64
    
  • TODO: init scripts

DrQueueOnRails configuration

  • Setup database:
    # mysql -p
    Enter password: 
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 39
    Server version: 5.1.41-3ubuntu12.6 (Ubuntu)
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    mysql> create database DrQueueOnRails_development;
    Query OK, 1 row affected (0.01 sec)
    
    mysql> grant usage on *.* to drqueueonrails@localhost identified by 'DQOR_PASSWORD';
    Query OK, 0 rows affected (0.00 sec)
    
    mysql> grant all privileges on DrQueueOnRails_development.* to drqueueonrails@localhost;
    Query OK, 0 rows affected (0.00 sec)
    
    mysql> quit
    Bye
    
  • Setup special user, database and install software on master:
    useradd -m -s /bin/bash drqueueonrails
    su drqueueonrails -
    cd
    git clone https://ssl.drqueue.org/git/DrQueueOnRails.git
    cd DrQueueOnRails
    cp config/database.yml.example config/database.yml
    nano config/database.yml
       development:
         adapter: mysql
         encoding: utf8
         reconnect: false
         database: DrQueueOnRails_development
         pool: 5
         username: drqueueonrails
         password: DQOR_PASSWORD
         host: localhost
    cp config/environment.rb.example config/environment.rb
    nano config/environment.rb
       ENV['DRQUEUE_MASTER'] ||= '127.0.0.1'
       ENV['WEB_PROTO'] ||= "http" 
       ENV['AVAIL_RENDERERS'] ||= "blender" 
    rake db:migrate
    
  • run DrQueueOnRails:
    su drqueueonrails -
    cd ~/DrQueueOnRails
    script/server
    
  • TODO: Apache / Nginx with Passenger, HTTPS

DrQueueCloudControl configuration

  • Setup special user and install software on master:
    useradd -m -s /bin/bash drqueuecloudcontrol
    echo "drqueuecloudcontrol ALL=(ALL) NOPASSWD:/home/drqueuecloudcontrol/DrQueueCloudControl/\
    generate_vpn_client_cert.sh" >>/etc/sudoers
    chmod 660 /home/drqueueonrails/DrQueueOnRails/db/DrQueueOnRails_development.sqlite3
    adduser drqueuecloudcontrol drqueueonrails
    su drqueuecloudcontrol -
    cd
    echo "export RUBYOPT=rubygems" >>.bashrc
    git clone https://ssl.drqueue.org/git/DrQueueCloudControl.git
    cp config.rb.example config.rb
    nano config.rb
        @pool_types = ['blender']
        @db_dqor_name = "DrQueueOnRails_development" 
        @db_dqor_pw = "DQOR_PASSWORD" 
        ENV['DRQUEUE_MASTER_FOR_VMS'] ||= '10.1.0.1'
        ENV['EC2_SLAVE_AMI'] ||= 'ami-99999999'
        ENV['EC2_KEY_NAME'] ||= 'mykeypair'
        ENV['EC2_INSTANCE_TYPE'] ||= 't1.micro'
        ENV['EC2_AVAIL_ZONE'] ||= 'eu-west-1a'
        ENV['EC2_SEC_GROUP'] ||= 'mysecgroup'
    
  • run DrQueueCloudControl:
    su drqueuecloudcontrol -
    cd ~/DrQueueCloudControl
    ruby dqcc.rb
    

Slave bootstrapping/provisioning