Razor-Server Chef Broker

Since razor has been re-written to razor-server, I had to re-write Chef broker as well. I modeled (copied) the new Chef broker after the original one as much as I could. There were, however, some natural differences and some new challenges. The biggest issue was that since the broker configuration JSON is stored as one string in the database, I needed to pass the validation file as one line and re-create valid file format on the broker hand off.

What Does It Do

The Chef broker is run after the post-install part of the razoring process. Once post-install is complete, the broker installation is invoked. The install.erb file for the appropriate broker is called, which contains some bash scripting to install all the needed bits. Chef broker installs chef-client on the new node, creates /etc/chef/validation.pem, /etc/chef/client.rb, and /etc/chef/first-boot.json files. In the end, the content of the first-boot file (contains run list) is executed by calling the following command:

chef-client -j /etc/chef/first-boot.json -E <%= broker.environment %>

The new Chef broker still expects the same inputs as the original broker. Let’s take a look at a sample json file for Chef broker (the new razor really likes JSON. JSON all the things!).

    "name": "openstack_chef_broker",
    "configuration": {
        "install_sh": "http://opscode.com/chef/install.sh",
        "version_string": "11.4.4",
        "chef_server_url": "https://chef.example.com:443",
        "validation_client_name": "chef-validator",
        "run_list": "role[allinone]",
        "environment": "openstack",
        "chef_client": "chef-client",
        "validation_key": "MIIok0h+mSHr1Pbi+v2950H1/7mzd71hXAcRHbmrdytAR+RjgtyibkkZTdSxjLP6o2qEuji0I7DL8cckHG56AvuAKhYoadu+9J/2ahmYr18CSqOmg4Sbh6CPm5edpqaAVTbE3Ec+1wN0IMl8KWtmGCrjpXzH1MDdaLZpIIYqC9DUVgLbd/i61hbiiYlky5wPdIKlKwilRl7alfsGKTUrVq1DuYiIZsDYrnYoLOERbzZU6yUxWSIasfVPJGpT9LvstChFjyjv/73etmhXwwIDAQABAoIBAHH1p1upllVJNMSAhSmLZfTe6Q9nT8unRFH1egcsni8dPXYyVzDQ1ztV3RFNDLjP01ZThqkjoA4TnkHvRnC90yR049eQNxn+7pctcTNW61aAglomMhBcFt7LlcDiiXfD3dVhoIDv4ijpfsNCDvEVain9Dv+krjGVP+dWKLKr2azZbrAnyJKJjKb8DkwFET7UADFgUiCwJCW7RyAAxL40WE1hyYGq03cum5+M+2MGTcCiiAumqIRjYyTioNl73zf/ckBKRBNb6DkVGa1drAwXqMtA0MAaJlQBoJJF9SehSf7rAKVMvQ+esS9kNXZhECgYEA28FKbU3K4mU2khZ8lJgomfelWV48sjeIH3OEuLK7WdKg2LAnNajiLAOHYi0egKiMJPnNyZ8uvOY0JABLHQXv10QKaNlHGqy+aRiRSb1Ya6SlbCg8hMoT29yWGa8k7AclLJSn20/dg0GAEQtegi/k3EEZq0CgYEA148QtVh4ng80d9P81rJEDkraAkKUNJPHBSKFqo1rNwSDsyxxTEXIzZSPYS1mjsdB4IxraiNm3WDhb+EldcSlXvmBDjYJck1WyLCw1JuyAS2VCEtdc38aPWGYZ157decuzkR8CU2TKw48KGSRTtwL8yYqTknqpAmnqf/KkWfVNi8CgYEA1a0AnX+Cwtf/U9UhlarN78fosxj6k5+DcHF2n9DKcvBnDctalMZ+BKX5wfB1NEyu2FU9T8rEO1DragYPA01+hCXYqVJ73OgSzUXiH31IuIID9u/0LyseTWOWoIxWJzdTh44xJ+wJlwNuYXxgjgVGaVZk2niXWTLxtTNEdCz0RpECgYAlholhGIq+8WSv656bfaMtXciAFjkYwhUmhrEAVOgyRr3qpjT/EzL23wLq5u1ws617OtbEPm984I2+XVKZIuerFgJqh+uzE1WlUGUoTgZ6AAZu0DfvkFPwFZpjfGY/y0QxsmhpcjDJkQvV+FP3h4UpCh7ZTDLl5axjgt0v3QSYDwKBgFde/5TO8N8U6lHr1YX+yY8wbQ9sVWPU8qruL5Qx11an1tm9Ja1Wbg8Sn0/A7h7Y331V4cDmVraUreULiTQwSO7N26IxQ3Rg/MQG3szUgP0MWYmjuG0c8zaFB73rnBpZ8xakF/xcRTt2Pb62dkw1VqFhzNc50bN+QvGmtEosIB9z"

    "broker-type": "chef"

As you can see, the validation file is on one line, as there is no good way to store formatted files in the database, and creating valid JSON with valid new line characters is also not easy. The whole “configuration” block actually gets flattened to a single line in the database, and as such, parsing must occur at some point. I chose the path of least resistance, one that does not require user to create a valid json from the validation file, rather just strip out new line characters.

Let’s take a look at each line in the configuration file:

"name": "openstack_chef_broker"

Is a name of the broker to be used, and it can be pretty much what you like.

"configuration": {

Is the beginning of configuration for the Chef itself.

"install_sh": "http://opscode.com/chef/install.sh",

This is the omnibus installer to be used for installing Chef on your newly installed server. If you have a closed network or have your own installer, it is likely that you will have a custom installer.

"version_string": "11.4.4"

Chef version to be installed.

"chef_server_url": "https://chef.example.com:443",

This is the URL of YOUR Chef server.

"validation_client_name": "chef-validator",

Client name to be used in node’s communication with the Chef server.

"run_list": "role[allinone]",

Initial run list. This is what is going to run after the chef client is installed on the new node.

"environment": "openstack",

Chef environment to be used in the initial Chef run on the new node.

"chef_client": "chef-client",

Binary name for the Chef client installed on the new node. If custom installer installed chef client in a special place or with a special name, specify that here (example: /usr/local/bin/chef-client)


The validation.pem file. This file must be turned into a single line, and the broker will construct it into it’s original form. Let me know if you have ideas on how to do this better.

"broker-type": "chef"

If you want a Chef broker, you must specify “chef” here. This is to tell razor-server that that’s the broker to use after the node install.


First, create broker.json file to contain the json:

    "name": "openstack_chef_broker",
    "configuration": {
        "install_sh": "http://opscode.com/chef/install.sh",
        "version_string": "11.4.4",
        "chef_server_url": "https://chef.example.com:443",
        "validation_client_name": "chef-validator",
        "run_list": "role[allinone]",
        "environment": "openstack",
        "chef_client": "chef-client",
        "validation_key": "MIIok0h+mSHr1Pbi+v2950H1/7mzd71hXAcRHbmrdytAR+RjgtyibkkZTdSxjLP6o2qEuji0I7DL8cckHG56AvuAKhYoadu+9J/2ahmYr18CSqOmg4Sbh6CPm5edpqaAVTbE3Ec+1wN0IMl8KWtmGCrjpXzH1MDdaLZpIIYqC9DUVgLbd/i61hbiiYlky5wPdIKlKwilRl7alfsGKTUrVq1DuYiIZsDYrnYoLOERbzZU6yUxWSIasfVPJGpT9LvstChFjyjv/73etmhXwwIDAQABAoIBAHH1p1upllVJNMSAhSmLZfTe6Q9nT8unRFH1egcsni8dPXYyVzDQ1ztV3RFNDLjP01ZThqkjoA4TnkHvRnC90yR049eQNxn+7pctcTNW61aAglomMhBcFt7LlcDiiXfD3dVhoIDv4ijpfsNCDvEVain9Dv+krjGVP+dWKLKr2azZbrAnyJKJjKb8DkwFET7UADFgUiCwJCW7RyAAxL40WE1hyYGq03cum5+M+2MGTcCiiAumqIRjYyTioNl73zf/ckBKRBNb6DkVGa1drAwXqMtA0MAaJlQBoJJF9SehSf7rAKVMvQ+esS9kNXZhECgYEA28FKbU3K4mU2khZ8lJgomfelWV48sjeIH3OEuLK7WdKg2LAnNajiLAOHYi0egKiMJPnNyZ8uvOY0JABLHQXv10QKaNlHGqy+aRiRSb1Ya6SlbCg8hMoT29yWGa8k7AclLJSn20/dg0GAEQtegi/k3EEZq0CgYEA148QtVh4ng80d9P81rJEDkraAkKUNJPHBSKFqo1rNwSDsyxxTEXIzZSPYS1mjsdB4IxraiNm3WDhb+EldcSlXvmBDjYJck1WyLCw1JuyAS2VCEtdc38aPWGYZ157decuzkR8CU2TKw48KGSRTtwL8yYqTknqpAmnqf/KkWfVNi8CgYEA1a0AnX+Cwtf/U9UhlarN78fosxj6k5+DcHF2n9DKcvBnDctalMZ+BKX5wfB1NEyu2FU9T8rEO1DragYPA01+hCXYqVJ73OgSzUXiH31IuIID9u/0LyseTWOWoIxWJzdTh44xJ+wJlwNuYXxgjgVGaVZk2niXWTLxtTNEdCz0RpECgYAlholhGIq+8WSv656bfaMtXciAFjkYwhUmhrEAVOgyRr3qpjT/EzL23wLq5u1ws617OtbEPm984I2+XVKZIuerFgJqh+uzE1WlUGUoTgZ6AAZu0DfvkFPwFZpjfGY/y0QxsmhpcjDJkQvV+FP3h4UpCh7ZTDLl5axjgt0v3QSYDwKBgFde/5TO8N8U6lHr1YX+yY8wbQ9sVWPU8qruL5Qx11an1tm9Ja1Wbg8Sn0/A7h7Y331V4cDmVraUreULiTQwSO7N26IxQ3Rg/MQG3szUgP0MWYmjuG0c8zaFB73rnBpZ8xakF/xcRTt2Pb62dkw1VqFhzNc50bN+QvGmtEosIB9z"

    "broker-type": "chef"

Create broker on the command line:

razor create-broker --json broker.json

Then, create the policy.json file:

  "name": "ubuntu_one",
  "repo": { "name": "ubuntu_server" },
  "task": { "name": "ubuntu" },
  "broker": { "name": "openstack_chef_broker" },
  "enabled": true,
  "hostname": "host${id}",
  "root_password": "secret",
  "max_count": "20",
  "rule_number": "107",
  "tags": [{ "name": "ubuntu_small", "rule": ["=", ["num", ["fact", "processorcount"]], 1]}]

Create policy:

razor create-policy --json policy.json

Now, your policy is set to use the new Chef broker.
Happy installing!


New razor-server vs old razor and what does the rewrite mean to you

If you have not been following google groups and other discussions, this is a summary of the main differences between the original razor and razor-server. The new razor-server is a somewhat different beast than the original razor. If you go to the original git repo, https://github.com/puppetlabs/razor, you will find a note stating “TL;DR: Razor is being rewritten.”

I will not go into the politics of why razor is being rewritten, the google groups provide the official explanation here: https://groups.google.com/forum/#!topic/puppet-razor/q4uCVMmUop0

The main differences are in installation and usage, most things are different, and the new razor-server pretty much carried the name only.


When it comes to the technologies used, the new razor-server has gotten rid of node.js components and MongoDB. It is now based on JRuby, and, as the name describes, is a server. The server component is TorqueBox, which in turn is based on JBoss. I have not tried it, but I think this will allow razor-server to be more scalable in the future in terms of deployment size. Database used is PostgreSQL, and the different modes now allow for dev/test/prod instances to run on one server.


Razor-server still uses a microkernel, however, the new MK is based on Fedora rather than Tiny Core Linux. Naturally, this makes the new MK much larger.

State Machine

One of the main things in the original razor was the state machine, which would let you know what state your nodes were in at any given time. At the time of writing, there is no easy way to accomplish the same in the razor-server, but there have been discussions about it, so who knows what will happen in the future.


Original server would collect facts about hardware using lshw. Razor-server does not utilize lshw, and the list of facts at the moment is rather limited. If you need special info about the hardware, you may need additional patches written for the razor-server.


The original razor ran on command line and as a razor service. Razor-server, on the other hand, runs in a JVM based application server, and all interactions with it are based through it is API. There is no need to curl all commands by hand however, since there is also a razor-client available. Razor-client is a command line tool to interact with the razor-server. The one annoyance with razor-client I have that it is super slow if it is using jruby. This means that you have to manage multiple ruby versions and take care that the client is not using jruby while the razor-server is.

In my opinion, razor-server is a little easier to use when it comes to configuring installers (also known as recipes or tasks) and policies, since everything can be passed as a json file instead of multiple commands on the command line.


Razor-server kept the concept of brokers, and currently supports puppet and chef brokers. No broker is still an option. Anybody can also add their own custom broker, however, there is a slight difficulty. Everything in the broker configuration will have to be stored as one entry in the PostgreSQL database. This is a problem if you need to have formatted files transmitted via the broker to the installed server. I had to solve this issue while writing the new Chef broker.


There is no path of upgrade between the versions of Razor and it is very unlikely that there will be one. These are essentially two different tools, both have great parts and the parts that are so-so.



Indexing Project Gutenberg with MongoDB and Elasticsearch

I am a big fan of Project Gutenberg. I love reading, and I love technology, and Project Gutenberg certainly combines the two! So when I needed some test data for my MongoDB integration with Elasticsearch, I downloaded all the plain text books from project’s FTP site. There is a lot more data available on the site, but I was interested only in the actual books and not in the various book formats for this project. Once I had all the books, I put all of them into a MongoDB database, connected it to Elasticsearch, and indexed all of them.

At the end of this little project, I had 37 GB worth of text files, 35 GB Elasticsearch index, and close to 78K files, all searchable in milliseconds. Some books came in multiple files or had several versions, so the actual number of files is much higher than the number of books. Project Gutenberg offers 43,856 free ebooks to download.

I am sure there are much better book content search tools out there, and this little project is merely a demonstration of MongoDB and Elasticsearch integration, rather than an actual useful search of all the books available at Project Gutenberg, though it certainly could be turned into one with a little extra work.

This is what I did. First, get all the books! If you do this, it may take a while. You will be downloading a lot of books. About 37 GB worth of them at the time of this writing.

wget -nd -r -l 10 -A.txt ftp://ftp.ibiblio.org/pub/docs/books/gutenberg/

This will put all the txt files in one directory, rather than build Project Gutenberg’s filing system.

After all the books were available locally, I read them into MongoDB with a Ruby script available here: https://github.com/eglute/fun-with-gutenberg/blob/master/guten_mongo.rb. The biggest issue was Ruby encoding. Each book had it’s own encoding style, and I needed to turn them all into a valid encoding for MongoDB. This I think was related more to the Ruby driver for MongoDB rather than MongoDB itself.

Since some books were larger than allowed MongoDB BSON document size, books got stored as GridFS objects. To learn more about GridFS, check out this website: http://docs.mongodb.org/manual/core/gridfs/.

Once all the books were in MongoDB, it was time to index them with Elasticsearch. Elasticsearch plugins are called rivers, and I used a MongoDB river for indexing: https://github.com/richardwilly98/elasticsearch-river-mongodb.

First, configure the river:

curl -XPUT "localhost:9200/_river/mongogridfs/_meta" -d'
   "type": "mongodb",
     "mongodb": {
       "db": "gutenberg",
       "collection": "fs",
       "gridfs": true
     "index": {
       "name": "gutenberg",
       "type": "files"

Note that both my MongoDB mongos process and Elasticsearch were on the same server, so no special configuration was needed. On the largest Rackspace cloud server instance, indexing took about an hour for all the books.

Once the books were indexed, I wrote another simple Ruby script for searching: https://github.com/eglute/fun-with-gutenberg/blob/master/guten_search.rb. Ruby client for Elasticsearch certainly makes it very easy to connect to the index as well as search. The script searches Elasticsearch index. If there are any hits, get the related MongoDB objects based on object id. Here is a sample subset of output:

ElasticSearch + MongoDB Based Project Gutenberg Word Search

It took 22 ms to search 77931 books for words 'Euroclydon'!
Found 53 results

Book: How Spring Came in New England
By: Charles Dudley Warner
Release Date: March, 2002  [Etext #3131]
Language: English, original file name: cwsne10.txt

Book:  Moby Dick; or The Whale
By:  Herman Melville
Release Date:  
Language: , original file name: moby11.txt

Book: From Edinburgh to India & Burmah
By: William G. Burn Murdoch
Release Date: September 24, 2007 [EBook #22749]
Language: English, original file name: 22749-8.txt

Right now, the script will search for all the words individually, so it is not terribly useful. Perhaps one day the script will support actual phrase searching or something more interesting. Ruby client for Elasticsearch certainly makes it very easy to connect to the index as well as search.

Thanks for reading!




Razor Chef Broker and CentOS

If you are trying to use razor to install CentOS and are running issues with the chef broker hand off, it is likely that this failure is cause the clock difference between your chef client and chef server. To verify, after install of OS is complete, log into your new node and run chef-client.

[root@centos9 ~]# chef-client
 Starting Chef Client, version 11.4.4
 [2013-08-23T07:42:02-06:00] INFO: *** Chef 11.4.4 ***
 [2013-08-23T07:42:02-06:00] INFO: [inet6] no default interface, picking the first ipaddress
 Creating a new client identity for centos9.razor.one using the validator key.
 [2013-08-23T07:42:03-06:00] INFO: Client key /etc/chef/client.pem is not present - registering
 Chef encountered an error attempting to create the client "centos9.razor.one"
Authentication Error:
 Failed to authenticate to the chef server (http 401).
 The request failed because your clock has drifted by more than 15 minutes.
 Syncing your clock to an NTP Time source should resolve the issue.

Once the server is installed, fixing this issue is simple. Log in to the server, install ntpdate, and correct the date:

yum install ntp
ntpdate pool.ntp.org

running chef-client again will be succesful.

However, this is not good enough for automated installs, it would be great if Razor did this to begin with. To correct it, edit file https://github.com/puppetlabs/Razor/blob/master/lib/project_razor/model/redhat/6/os_boot.erb

Before adding puppetlabs repo, add these two lines:

yum -y install
ntpntpdate pool.ntp.org

This should be in a patch on github, but it appears that no recent patches have been accepted to the current razor code.


What is your favorite security question?

There is only one simple answer to this question- none of your pre-filled “What is your favorite _______ ?” will be appropriate for me. The problem with these questions is that they are either hard to answer for me, or too easy for you to find out by checking out my facebook/twitter/linkedin/whatever social networks. Please, if you are in charge of creating these “security” questions, leave them blank for me to fill in!

If I am forced to pick out of 8 or 10 bad questions, I will end up either making them really easy to answer, or forget the answers to them. Just this evening I was on the phone for over 30 min waiting to reset my account on the system where I could not correctly answer one of these “secure” questions. Where did I spend my childhood summers? Well, I do know where I spent them, but I could not remember which one of the names/places I would have picked a year ago when I signed up for this super secure service!

After reseting my password over the phone, I went into my profile and tried to change the answers. I had to have 3 questions, and these were some of the choices:

Mother’s maiden name- probably the least secure type of question a system can ask. My mother may not have her maiden name on her facebook profile, but yours might.

What is your high school mascot? You could probably find out if you know anything at all about me.

What is your favorite teacher’s name? Seriously? I had a few teachers I really liked, but would be hard pressed to pick one. The answer would be, “it depends”.

What is your favorite movie? Now, I know there are a lot of people out there that have their one and only favorite movie that they have memorized. I actually believe my favorite movie might change over time. What if I see my favorite movie this coming weekend?

What is your favorite show? The same problem as with the favorite movie. Is it all-time favorite? Or is it current season favorite?

You get the idea. Favorite this, favorite that. Best friend, best vacation, best anything. Pet’s name. Either too easy, or too hard. Some people share a lot about themselves online. So please, let me fill in my own security question if that’s the route you pick for making your system secure. Please.




DevStack and Ceph Tutorial

I just started looking at Ceph, and wanted to set it up to be used with DevStack. Ceph website has great documentation, so if you are looking on how to setup OpenStack and Ceph for production use, check it out! However, if you just want to get the DevStack with Ceph running, follow along this post.

I started out following directions on how-to Ceph and DevStack on EC2, but since I am using a Rackspace Cloud Server, some of the instructions were not relevant to me. Also, since everything is running on one server, I am not using  Ceph auth.

Install DevStack

The server: Ubuntu 12.04, 4GB RAM. After creating it through the control panel, login, do the usual updates, and create a new user. DevStack will create a user for you, but I find it that things run much smoother if I create the user myself.

apt-get update
apt-get install git
groupadd stack
useradd -g stack -s /bin/bash -d /opt/stack -m stack

In sudo-ers file, add this line:


This will allow stack user to use sudo without password.

sudo su stack

As a stack user, install DevStack:

git clone git://github.com/openstack-dev/devstack.git
cd devstack

Create localrc file:



Run the install script:


Go get some coffee, and when all the scripts complete, login to Horizon, and make sure creating instances is working (create one!).

Install Ceph

Now that devstack is installed, it is time to install Ceph. I followed the “5 Minute Quick Start” with a couple small changes and omissions.

wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -
echo deb http://ceph.com/debian/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt-get update && sudo apt-get install ceph

Create Ceph configuration file:

sudo vi /etc/ceph/ceph.conf

and add the following:


# For version 0.55 and beyond, you must explicitly enable
# or disable authentication with "auth" entries in [global].

auth cluster required = none
auth service required = none
auth client required = none

osd journal size = 1000

#The following assumes ext4 filesystem.
filestore xattr use omap = true
# For Bobtail (v 0.56) and subsequent versions, you may
# add settings for mkcephfs so that it will create and mount
# the file system on a particular OSD for you. Remove the comment `#`
# character for the following settings and replace the values
# in braces with appropriate values, or leave the following settings
# commented out to accept the default values. You must specify the
# --mkfs option with mkcephfs in order for the deployment script to
# utilize the following settings, and you must define the 'devs'
# option for each osd instance; see below.

#osd mkfs type = {fs-type}
#osd mkfs options {fs-type} = {mkfs options} # default for xfs is "-f"
#osd mount options {fs-type} = {mount options} # default mount option is "rw,noatime"

# For example, for ext4, the mount option might look like this:

#osd mkfs options ext4 = user_xattr,rw,noatime

# Execute $ hostname to retrieve the name of your host,
# and replace {hostname} with the name of your host.
# For the monitor, replace {ip-address} with the IP
# address of your host.


host = {hostname}
mon addr = {IP}:6789

host = {hostname}

# For Bobtail (v 0.56) and subsequent versions, you may
# add settings for mkcephfs so that it will create and mount
# the file system on a particular OSD for you. Remove the comment `#`
# character for the following setting for each OSD and specify
# a path to the device if you use mkcephfs with the --mkfs option.

#devs = {path-to-device}

host = {hostname}
#devs = {path-to-device}

Note that you will need to change the hostname and IP address to match your own. If you want to use Ceph auth, you will need to change “none” to “cephx”.

Create directories for Ceph daemons:

sudo mkdir -p /var/lib/ceph/osd/ceph-0
sudo mkdir -p /var/lib/ceph/osd/ceph-1
sudo mkdir -p /var/lib/ceph/mon/ceph-a
sudo mkdir -p /var/lib/ceph/mds/ceph-a

Deploy Ceph, generate user key, start Ceph:

cd /etc/ceph
sudo mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring
sudo service ceph -a start
sudo ceph health

When “ceph health” command returns “HEALTH_OK”, Ceph is ready to be used. Create volumes:

ceph osd pool create volumes 128
ceph osd pool create images 128

Install client libraries:

sudo apt-get install python-ceph

Setup pool permissions, create users and keyrings:

ceph auth get-or-create client.volumes mon 'allow r' osd 'allow rwx pool=volumes, allow rx pool=images'
ceph auth get-or-create client.images mon 'allow r' osd 'allow rwx pool=images'
sudo useradd glance
sudo useradd cinder
ceph auth get-or-create client.images | sudo tee /etc/ceph/ceph.client.images.keyring
sudo chown glance:glance /etc/ceph/ceph.client.images.keyring
ceph auth get-or-create client.volumes | sudo tee /etc/ceph/ceph.client.volumes.keyring
sudo chown cinder:cinder /etc/ceph/ceph.client.volumes.keyring

Edit glance configuration file:

sudo vi /etc/glance/glance-api.conf
default_store = rbd

[ ... ]

# ============ RBD Store Options =============================

# Ceph configuration file path
# If using cephx authentication, this file should
# include a reference to the right keyring
# in a client. section
rbd_store_ceph_conf = /etc/ceph/ceph.conf

# RADOS user to authenticate as (only applicable if using cephx)
rbd_store_user = images

# RADOS pool in which images are stored
rbd_store_pool = images

# Images will be chunked into objects of this size (in megabytes).
# For best performance, this should be a power of two
rbd_store_chunk_size = 8

Add the following lines to the cinder configuration:

sudo vi /etc/cinder/cinder.conf

Configure DevStack to Use Ceph

At this point, both Ceph and DevStack are configured, however, since the configuration files for glance and cinder were changed, restart all glance and cinder services.

To restart the services in DevStack, re-join screen, bring up services, quit and restart them.
Rejoin screen:

screen -r

If you get something like this:

Cannot open your terminal '/dev/pts/0' - please check.

This means you do not have permissions to it. Simple fix:

sudo chmod 777 /dev/pts/0

After re-joining screen, type control-a keys, followed by ” (quotation mark). This will present you with all the running services:

Num Name                    Flags

  0 shell                       $
  1 key                      $(L)
  2 horizon                  $(L)
  3 g-reg                    $(L)
  4 g-api                    $(L)
  5 n-api                    $(L)
  6 n-cond                   $(L)
  7 n-cpu                    $(L)
  8 n-crt                    $(L)
  9 n-net                    $(L)
 10 n-sch                    $(L)
 11 n-novnc                  $(L)
 12 n-xvnc                   $(L)
 13 n-cauth                  $(L)
 14 n-obj                    $(L)
 15 c-api                    $(L)
 16 c-vol                    $(L)
 17 c-sch                    $(L)

Use up and down arrows to select the service for restart, and click enter. Type control-c, to stop the service, then up-arrow once to bring up the previous command, and enter to start it up again. Rinse and repeat.

Services that need to be restarted:

3 g-reg                    $(L)
  4 g-api                    $(L)
 15 c-api                    $(L)
 16 c-vol                    $(L)
 17 c-sch                    $(L)

If you are looking on some general DevStack, logging, and screen tips, check out this blog: http://vmartinezdelacruz.com/logging-and-debugging-in-openstack/

Use Ceph

Now your DevStack will use Ceph! Go to Horizon. Under Project/Volumes, create a couple new volumes and attach them to your VM. On command line, list volumes:

rbd ls -p volumes

You should see something similar to this:


This is all you should need to get going with Ceph and DevStack.

Useful resources:

DevStack: http://devstack.org/
Ceph: http://ceph.com/docs/master/
Ceph + DevStack on EC2: http://ceph.com/howto/building-a-public-ami-with-ceph-and-openstack/
Ceph 5 Minute Quick Start: http://ceph.com/docs/master/start/quick-start/
Ceph and OpenStack: http://ceph.com/docs/master/rbd/rbd-openstack/
Some good DevStack logging and screen tips: http://vmartinezdelacruz.com/logging-and-debugging-in-openstack/

Have fun with!


The Making of Cloud of USB Sticks

This is a post of how helpful people are in the OpenStack community. If you were to visit the lobby of Hilton Executive tower in Portland last Wednesday, you would have seen me and a couple other people making the cloud of USB sticks.

Last Thursday, I gave my first ever conference talk/tutorial at the OpenStack Summit. The goal of my tutorial was to show how to use Razor, and how easy it is to install OpenStack with it. There were a couple issues I had to solve for this tutorial: what is the fastest way to get people to have a running Razor on their system and how to do so over not very reliable internet connection.

To solve the first issue, I created a VM that had all the necessary components installed on it, everything from Razor to Chef server. Just to give some perspective, it takes just over an hour to download and setup all the major components, including loading cookbooks for OpenStack installation and loading them to the Chef server. Considering that my talk needed to be under 1 hour, live install would not have been a good option.

After I got to the conference, I realized that there is no way I can share my 3.6 GB VM over the intertubes! I probably should have timed downloading the VM over the regular (not on infinite work bandwidth) internet, to realize that it was a bad idea anyways. Luckily for me, I got some help. First, I talked to the Rackspace Private Cloud training team, and they provided me with wireless routers for setting up local network for sharing VM over internal network. The day of the talk, super helpful @Thediopter actually configured the routers for me, in record time! This local network allowed me to serve the file off my laptop while I was giving the talk.

Since I was not sure wireless local network was going to work, I decided that I needed to put my VM on USB sticks. I knew that eNovance is giving out USB drives at their vendor booth. I shared my problem with them, and they happily supplied me with a lot of USB drives. Going by the Suse booth, I noticed that they too had USB drives, and I asked if I could have some for my talk. They gladly supplied me with additional drives.

By now, I had 45 drives, and a limited amount of time to copy my VM to them. Surely there is a better way to copy something to a USB drive than one at the time. As I was walking by the Piston booth, I was asked about the USB sticks. I repeated my story of needing to make lots of USB sticks, and what do you know? I walked away with a 10 port USB hub!

Back at the hotel lobby, the baking of the USB sticks began. My coworkers helped me out with a little script to run on my mac to copy, and all I have to say, I am not sure where I would have been without them!

So, if you ever find yourself with a mac, USB hub, and lots of USB drives, this is how you copy to them on a command line. First, make sure you are root, then:

for i in `jot 10 2`; do asr --noverify --erase --noprompt --source /dev/disk1s1 --target /dev/disk${i}s1 & done

What this script is doing: copying from /dev/disk1s1 to all the other disks (USB drives), erasing destination, and does not verify anything. This process may take a while, but it is still faster than copying one at the time.

So, thanks again to everyone that helped out in the making of the USB cloud!


Curious what was on the USB drives? This.


Getting Started Using Razor and Chef (Razor Appliance)

Instead of having everyone to install Razor to demo it during the OpenStack Summit in Portland, I created a Razor appliance. Installing and configuring Razor is not difficult, but could be a little time consuming if you do not have either Puppet or Chef already set up and configured. You can, of course, install it manually or go the Vagrant route: https://github.com/blueboxgroup/razor-vagrant-lab.

Slides for the OpenStack Summit tutorial can be found here: http://www.slideshare.net/mattray/bare-metal-to-openstack-with-razor-and-chef (thanks @mattray)

Razor VM

For creating Razor VM, I used Ubuntu 12.04 server. It has the following installed and configured:

Razor (installed using chef: https://github.com/bbg-cookbooks/razor)
TFTP server
DHCP server
Chef server (loaded with Rackspace Private Cloud cookbooks https://github.com/rcbops/chef-cookbooks)
MicroKernel and Ubuntu server ISO loaded

The VM needs to have 2 nics, one for internal, and one for external traffic.

Download VM from here (3.6 GB download): https://ushnishtha.files.wordpress.com/razor.tar and un-tar it.

Load it either into VMware Fusion or VirtualBox. Please see bellow for special instructions for either!

Login, and switch to root:
username: anystacker
password: razor

Verify Razor is running:

  • root@razor:~# /opt/razor/bin/razor_daemon.rb status

If it is not running, start it up:

  • root@razor:~# /opt/razor/bin/razor_daemon.rb start

Now you can start using Razor. Here are some useful commands:

  • razor image
  • razor node
  • razor active_model
  • razor model
  • razor broker
  • razor policy

Running Razor is not very interesting, unless you try to create a new VM and watch Razor boot it. For this to happen, you will need to create a new VM in either Fusion or VirtualBox. The new VM needs to have a nic on internal network in order to be booted by razor. There are a couple special considerations, depending whether you are using VirtualBox or VMware Fusion.

VMware Fusion

While importing the appliance into Fusion, you may be presented with some errors/warnings. Dismiss them, it should work anyways.

If you are using Fusion, under Preferences/Network, make sure DHCP is disabled for the private network:


Razor appliance will need two nics, public and private. Make sure nic #1 is public:


Razor appliance nic #2 should be set to be on private network. Any subsequent VM to be booted by Razor should also be on private network:


Virtual Box

If you are using Virtual Box:

Make sure the extension pack is installed. To verify it, on command line type the following:

  • VBoxManage list extpacks

If you do not have extention pack, get it here:

When importing Razor appliance, accept all default settings, do not change anything. When creating a new VM to be booted by Razor, change boot order and set network to be internal.

For changing the boot order to include network boot, after creating a new VM, go to it’s Settings and under System, and select network boot order.


Once the boot order is set, change the VM network to be attached to internal network. It should look similar to this:


Now, once the new VM is powered on, it should start booting. If you are presented with an option to select a start up disk, cancel out of it. If Razor is listening on the right network, and your VM settings are correct, you should see something like this:


It should be followed by Razor Mikro Kernel loading information and then regular installation.


I did not try this myself, but @VMRandy was able to import the appliance into vSphere. First he uploaded it to Workstation, upgraded to hardware version 9, and then uploaded to vSphere. Once the VM was loaded and network configured, he was able to boot new nodes in vSphere using Razor.

If Things Don’t Work

Most likely, it is the network.

  • Check the settings, make sure Razor appliance has two nics, a public and a private.
  • Make sure there are no other DHCP servers affecting things and confusing your new VM.
  • Make sure the new VM is connecting on Private Network.
  • Make sure Razor appliance eth1 is listening on
  • Make sure Razor is running.
  • Make sure MongoDB is running.
  • If using Virtual Box, install extension pack.
  • If using Virtual Box, set boot order.

There are a few other things that could go wrong, but usually, it’s the network.

Everything Is Working, Now What?

Congratulations, you were able to download a large file from the internets, successfully imported into your favorite virtualization software, and deployed your first VM using Razor. If you list the Razor policies setup, you will notice that there are two of them. The second policy is for servers with two nics, and will install OpenStack for you on such servers. Go ahead and try- when creating a new server, make sure both nics are on private network. This time, spinning up a new server will take a little longer, as the Razor hands off the new node to the broker. The broker, in turn, will install chef client on it and run the initial run list. In our case, it is OpenStack installation.

Don’t stop at two servers! Try adding different policies and tags, and create servers with different configurations to match them. Check the “razor active_model” command, and see how it matches up with the “knife node list” command.

Have fun!


P.S.: about the making of USB sticks: https://anystacker.com/2013/04/the-making-of-cloud-of-usb-sticks/

P.S.S.: youtube video of the OpenStack Summit presentation: http://www.youtube.com/watch?feature=player_detailpage&v=Lal3aL5JH3o#t=183s


Razor Broker Custom Metadata

Ever wanted to know what information Razor collects about your system that you could lay your hands on? Wonder no more!

When handing of your newly created node to a broker, besides installing the client and registering it with the server, Razor also injects custom facts into the new node. They consist of two parts, “razor_metadata” and “razor_attributes”. Razor metadata is available for both chef and puppet brokers, and consist of the following:

razor_metadata: {
    razor_tags: "memsize_1GiB,nics_1,cpus_1,IntelCorporation,vmware_vm",
    razor_node_uuid: "JtFXS98553aTEsrUhy6V9",
    razor_active_model_uuid: "PQG5iXxTIDGgEeIQ9qsnD",
    razor_model_uuid: "7RdhL19S5pBAAWTqNldfSl",
    razor_model_name: "ubuntu_precise",
    razor_model_description: "Ubuntu Precise Model",
    razor_model_template: "linux_deploy",
    razor_policy_count: "14"

Razor attributes are a bit more extensive, and I believe are currently injected only into the chef broker. This very detailed information about your system is collected by the Razor microkernel, and can also be found under active model/node/@attributes_hash in the database. The data looks like this:

razor_attributes: {
    mk_hw_cpu0_size: "2200MHz",
    mk_hw_lscpu_CPU_MHz: "2199.513",
    mk_hw_mem_description: "System Memory",
    mk_hw_nic0_clock: "66MHz",
    mk_hw_cpu0_physical_id: "4",
    ipaddress_lo: "",
    mk_hw_fw_physical_id: "0",
    processorcount: "1",
    macaddress: "6A:8C:81:E3:6C:75",
    mk_hw_nic0_version: "01",
    mk_hw_disk0_size: "20GiB (21GB)",
    manufacturer: "VMware, Inc.",
    mk_hw_bus_version: "None",
    architecture: "i386",
    memorytotal: "1005.62 MB",
    mk_hw_nic_count: "1",
    mk_hw_disk_count: "1",
    mk_hw_lscpu_L2_cache: "256K",
    mk_hw_lscpu_CPU_sockets: "1",
    mk_hw_cpu0_slot: "CPU socket #0",
    network_lo: "",
    hardwareisa: "unknown",
    mk_hw_lscpu_Stepping: "7",
    mk_hw_fw_capabilities: "isa pci pcmcia pnp apm upgrade shadowing escd cdboot bootselect edd int5printscreen int9keyboard int14serial int17printer int10video acpi smartbattery biosbootspecification netboot",
    "mk_hw_lscpu_CPU_op-modes": "32-bit, 64-bit",
    mk_hw_nic0_width: "64 bits",
    mk_hw_cpu0_vendor: "Intel Corp.",
    processor0: "Intel(R) Core(TM) i7-2720QM CPU @ 2.20GHz",
    mk_hw_lscpu_Byte_Order: "Little Endian",
    netmask_eth0: "",
    mk_hw_fw_vendor: "Phoenix Technologies LTD",
    mk_hw_nic0_logical_name: "eth0",
    mk_hw_disk0_logical_name: "/dev/sda",
    domain: "dns.hosts",
    macaddress_dummy0: "6A:8C:81:E3:6C:75",
    boardserialnumber: "None",
    mk_hw_bus_vendor: "Intel Corporation",
    mk_hw_cpu0_capabilities: "boot fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss nx rdtscp x86-64 constant_tsc up arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm ida arat epb xsaveopt pln pts",
    mk_hw_mem_size: "1GiB",
    mk_hw_lscpu_L1i_cache: "32K",
    mk_hw_cpu0_serial: "0008-06A9-0187-0000-0000-0000",
    mk_hw_fw_size: "87KiB",
    network_eth0: "",
    hostname: "mk000C291F2C5C",
    mk_hw_lscpu_Model: "42",
    netmask_lo: "",
    mk_hw_nic0_capacity: "1Gbit/s",
    mk_hw_cpu0_product: "Intel(R) Core(TM) i7-2720QM CPU @ 2.20GHz",
    mk_hw_fw_description: "BIOS",
    ipaddress: "",
    physicalprocessorcount: 1,
    macaddress_eth0: "00:0B:29:1F:2D:5C",
    mk_hw_nic0_bus_info: "pci@0000:02:01.0",
    mk_hw_disk0_bus_info: "scsi@2:0.0.0",
    boardproductname: "440BX Desktop Reference Platform",
    mk_hw_bus_product: "440BX Desktop Reference Platform",
    mk_hw_cpu0_width: "64 bits",
    mk_hw_mem_slot: "System board or motherboard",
    mk_hw_lscpu_L1d_cache: "32K",
    mk_hw_nic0_configuration: "autonegotiation=on broadcast=yes driver=e1000 driverversion=7.3.21-k8-NAPI duplex=full firmware=N/A ip= latency=0 link=yes mingnt=255 multicast=yes port=twisted pair speed=1Gbit/s",
    mk_hw_cpu0_version: "6.10.7",
    mk_hw_fw_date: "07/02/2012",
    fqdn: "mk000C291F2C5C.dns.hosts",
    mk_hw_lscpu_CPU_family: "6",
    virtual: "vmware",
    mk_hw_nic0_size: "1Gbit/s",
    mk_hw_cpu0_description: "CPU",
    serialnumber: "VMware-56 4c e7 4d ae 3c 03 d5-4f e6 9e dc a4 1f 3c 6c",
    mk_hw_bus_physical_id: "0",
    hardwaremodel: "i686",
    mk_hw_lscpu_Architecture: "i686",
    mk_hw_nic0_physical_id: "1",
    boardmanufacturer: "Intel Corporation",
    mk_hw_bus_description: "Motherboard",
    mk_hw_disk0_physical_id: "0.0.0",
    mk_hw_cpu0_capacity: "4230MHz",
    mk_hw_lscpu_BogoMIPS: "4400.31",
    mk_hw_mem_physical_id: "e2",
    type: "Other",
    mk_hw_nic0_capabilities: "pm pcix bus_master cap_list rom ethernet physical logical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation",
    mk_hw_cpu0_bus_info: "cpu@0",
    is_virtual: "true",
    mk_hw_fw_version: "6.00",
    netmask: "",
    mk_hw_lscpu_Vendor_ID: "GenuineIntel",
    mk_hw_nic0_serial: "00:0c:27:2f:1c:7c",
    mk_hw_cpu_count: "1",
    ipaddress_eth0: "",
    productname: "VMware Virtual Platform",
    mk_hw_bus_serial: "None",
    mk_hw_nic0_description: "Ethernet interface",
    interfaces: "dummy0,eth0,lo",
    mk_hw_lscpu_L3_cache: "6144K",
    memorysize: "1005.62 MB",
    mk_hw_disk0_description: "SCSI Disk"

Once this data is injected into the chef node, you can access it just like any other attribute.


Razor Chef Broker, Updated

If you are looking for info on the razor-server broker, check this blog post: https://anystacker.com/2014/01/razor-server-chef-broker/

A few weeks ago i posted about the razor chef broker, while it was still a work in progress.  After some awesome collaboration with Fletcher Nichol, Tom McSweeney, and Cody Bunch, chef broker is now officially part of Razor.

Originally, while puppet broker was the only one available, the CLI options for adding a broker were geared towards it.  While working on chef razor, many more options were needed, so the whole CLI interaction for adding a broker changed. The new flow flows much nicer, in my opinion!

Here is a simple example of adding new razor broker, step by step.
Lets check available plugins, make sure that the chef is one of the available options:

root@ubuntu:/opt/razor# razor broker get plugins

Available Broker Plugins:
 Plugin Description
 puppet PuppetLabs PuppetMaster
 chef Opscode Chef

List current brokers:

root@ubuntu:/opt/razor# razor broker
 Broker Targets:
 Name Description Plugin UUID
 puppet puppet puppet 11fTD6ejTXmbS4nyIu0ZF5

Lets add a new broker (-p for plugin, -n for name, -d for description):

root@ubuntu:/opt/razor# razor broker add -p chef -n Chef -d Production
 --- Building Broker (chef):
Please enter the URL for the Chef server. (example: https://chef.example.com:4000)
 (QUIT to cancel)

Your chef server can be in any valid URL format, whether it be IP:port, or just IP. This is were your new client will be connecting to to call home.

 > https://chef.example.com:4000

What chef version do you want to use?

Please enter the Chef version (used in gem install). (example: 10.16.2)
(QUIT to cancel)
> 10.16.2

Chef client needs a server validation file. This generally can be found on your server under /etc/chef/validation.pem. Copy the whole file, paste, and add a blank line:

Please enter a paste of the contents of the validation.pem file, followed by a blank line. (example: -----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAA...) 
(QUIT to cancel)

How should your client auth against chef server? More info on chef auth can be found here: http://wiki.opscode.com/display/chef/Authentication+and+Authorization

Please enter the validation client name. (example: myorg-validator) 
default: chef-validator
(QUIT to cancel)
 > chef-validator

Which environment? More about Chef environments: http://wiki.opscode.com/display/chef/Environments

Please enter the Chef environment in which the chef-client will run. (example: production) 
default: _default
(QUIT to cancel)

This script will actually perform the installation. Replace with your own if you don’t trust the internets!

Please enter the Omnibus installer script URL. (example: http://mirror.example.com/install.sh) 
default: http://opscode.com/chef/install.sh
(QUIT to cancel)

More custom options, press enter for default:

Please enter an alternate path to the chef-client binary. (example: /usr/local/bin/chef-client) 
default: chef-client
(QUIT to cancel)

Optional run list, type “SKIP” if you don’t have any.

Please enter an optional run_list of common base roles. (example: role[base],role[another]) 
(SKIP to skip, QUIT to cancel)

Finally, you have a brand new chef broker, ready to be added to your policy:

 Name =>  Chef
 Description =>  Production
 Plugin =>  chef
 UUID =>  7HYCfiyr5LQ13ScnLoRlUB
 Chef Server URL =>  https://chef.example.com:4000
 Chef Version =>  10.16.2
 Validation Key MD5 Hash =>  6372aca0ebceddb65c4e934d35f789c2
 Validation Client Name =>  chef-validator
 Bootstrap Environment =>  _default
 Install Sh Url =>  http://opscode.com/chef/install.sh
 Chef Client Path =>  chef-client
 Base Run List =>

Now, when listing available brokers, you should see your new chef broker in the list:

root@ubuntu:/opt/razor# razor broker
Broker Targets:
 Name   Description  Plugin           UUID           
Chef    Production   chef    7HYCfiyr5LQ13ScnLoRlUB  
puppet  puppet       puppet  11fTD6ejTXmbS4nyIu0ZF5

You can view the details of the broker by specifying it’s UUID:

root@ubuntu:/opt/razor# razor broker 7HYCfiyr5LQ13ScnLoRlUB
 Name =>  Chef
 Description =>  Production
 Plugin =>  chef
 UUID =>  7HYCfiyr5LQ13ScnLoRlUB
 Chef Server URL =>  https://chef.example.com:4000
 Chef Version =>  10.16.2
 Validation Key MD5 Hash =>  6372aca0ebceddb65c4e934d35f789c2
 Validation Client Name =>  chef-validator
 Bootstrap Environment =>  _default
 Install Sh Url =>  http://opscode.com/chef/install.sh
 Chef Client Path =>  chef-client
 Base Run List =>  

Razor Facts

Besides installing chef client and registering it with the server, Razor also injects custom facts into the new node. I split them out into additional blog post, since there are a few of them!