Jan 202014
 

If you want to play with the new razor-server version but don’t feel like installing all the bits by hand, try out a little vagrant script I wrote at https://github.com/eglute/razor-server-vagrant. So far, it has been tested with the WMware Fusion, and is somewhat of a work in progress. Keep in mind, that it takes a very long time to set it up, as it downloads, installs and configures the following bits:

curl
git
dnsmasq
postgresql
java
JRuby
TorqueBox
razor-client
razor microkernel
ubuntu server image

Once the vagrant is up and running, you will have a fully functional razor-server, setup with a policy and ready to install ubuntu server to the VMs attached to the newly created private network. The base policy will look for VMs with 1 processor.

The Vagrantfile comes with some IPs baked in. If you do not like the IPs I have picked, change them. Please take care that the IPs are in appropriate ranges.

config.vm.provision "shell", path: "provision.sh", :args =>"172.16.2.137 172.16.2.150 172.16.2.255"

First IP, or $IP_ADDRESS will be the IP of razor-server on the private network created. Note that it will be on eth1.

Second IP, or $IP_RANGE is going to be passed into dnsmasq configuration file for the DHCP range upper limit. The lower limit is the $IP_ADDRESS.

The third IP, or the $IP_BROADCAST is the IP used for broadcast in the ntp.conf. In this case, we have our own ntp server because in some cases you may really need accurate time.

So, check out the code. Then, add a vagrant box:

vagrant box add precise_fusion http://files.vagrantup.com/precise64_vmware.box

Once that is done, you are ready to do vagrant up:

vagrant up --provider=vmware_fusion

Now, go and have some coffee while vagrant does all the work for you!

Once vagrant is done with all the hard work, you can do

vagrant ssh
sudo su -
razor

and use razor commands. They will be a bit slow since razor client is using JRuby instead of a proper version of ruby (the author was cutting corners).

To install a new VM in VMware Fusion, create a new custom VM, add a new network device and specify the newly created vmnet (look for the latest vmnet2 or vmnet3). If all went well, your new VM will do a pxe boot off of a razor, and you will have a new node with Ubuntu server installed.

Happy vagranting and razoring!

-eglute

 Posted by at 11:28 pm
Jan 202014
 

Since razor has been re-written to razor-server, I had to re-write Chef broker as well. I modeled (copied) the new Chef broker after the original one as much as I could. There were, however, some natural differences and some new challenges. The biggest issue was that since the broker configuration JSON is stored as one string in the database, I needed to pass the validation file as one line and re-create valid file format on the broker hand off.

What Does It Do

The Chef broker is run after the post-install part of the razoring process. Once post-install is complete, the broker installation is invoked. The install.erb file for the appropriate broker is called, which contains some bash scripting to install all the needed bits. Chef broker installs chef-client on the new node, creates /etc/chef/validation.pem, /etc/chef/client.rb, and /etc/chef/first-boot.json files. In the end, the content of the first-boot file (contains run list) is executed by calling the following command:

chef-client -j /etc/chef/first-boot.json -E <%= broker.environment %>
Input

The new Chef broker still expects the same inputs as the original broker. Let’s take a look at a sample json file for Chef broker (the new razor really likes JSON. JSON all the things!).

{
    "name": "openstack_chef_broker",
    "configuration": {
        "install_sh": "http://opscode.com/chef/install.sh",
        "version_string": "11.4.4",
        "chef_server_url": "https://chef.example.com:443",
        "validation_client_name": "chef-validator",
        "run_list": "role[allinone]",
        "environment": "openstack",
        "chef_client": "chef-client",
        "validation_key": "MIIok0h+mSHr1Pbi+v2950H1/7mzd71hXAcRHbmrdytAR+RjgtyibkkZTdSxjLP6o2qEuji0I7DL8cckHG56AvuAKhYoadu+9J/2ahmYr18CSqOmg4Sbh6CPm5edpqaAVTbE3Ec+1wN0IMl8KWtmGCrjpXzH1MDdaLZpIIYqC9DUVgLbd/i61hbiiYlky5wPdIKlKwilRl7alfsGKTUrVq1DuYiIZsDYrnYoLOERbzZU6yUxWSIasfVPJGpT9LvstChFjyjv/73etmhXwwIDAQABAoIBAHH1p1upllVJNMSAhSmLZfTe6Q9nT8unRFH1egcsni8dPXYyVzDQ1ztV3RFNDLjP01ZThqkjoA4TnkHvRnC90yR049eQNxn+7pctcTNW61aAglomMhBcFt7LlcDiiXfD3dVhoIDv4ijpfsNCDvEVain9Dv+krjGVP+dWKLKr2azZbrAnyJKJjKb8DkwFET7UADFgUiCwJCW7RyAAxL40WE1hyYGq03cum5+M+2MGTcCiiAumqIRjYyTioNl73zf/ckBKRBNb6DkVGa1drAwXqMtA0MAaJlQBoJJF9SehSf7rAKVMvQ+esS9kNXZhECgYEA28FKbU3K4mU2khZ8lJgomfelWV48sjeIH3OEuLK7WdKg2LAnNajiLAOHYi0egKiMJPnNyZ8uvOY0JABLHQXv10QKaNlHGqy+aRiRSb1Ya6SlbCg8hMoT29yWGa8k7AclLJSn20/dg0GAEQtegi/k3EEZq0CgYEA148QtVh4ng80d9P81rJEDkraAkKUNJPHBSKFqo1rNwSDsyxxTEXIzZSPYS1mjsdB4IxraiNm3WDhb+EldcSlXvmBDjYJck1WyLCw1JuyAS2VCEtdc38aPWGYZ157decuzkR8CU2TKw48KGSRTtwL8yYqTknqpAmnqf/KkWfVNi8CgYEA1a0AnX+Cwtf/U9UhlarN78fosxj6k5+DcHF2n9DKcvBnDctalMZ+BKX5wfB1NEyu2FU9T8rEO1DragYPA01+hCXYqVJ73OgSzUXiH31IuIID9u/0LyseTWOWoIxWJzdTh44xJ+wJlwNuYXxgjgVGaVZk2niXWTLxtTNEdCz0RpECgYAlholhGIq+8WSv656bfaMtXciAFjkYwhUmhrEAVOgyRr3qpjT/EzL23wLq5u1ws617OtbEPm984I2+XVKZIuerFgJqh+uzE1WlUGUoTgZ6AAZu0DfvkFPwFZpjfGY/y0QxsmhpcjDJkQvV+FP3h4UpCh7ZTDLl5axjgt0v3QSYDwKBgFde/5TO8N8U6lHr1YX+yY8wbQ9sVWPU8qruL5Qx11an1tm9Ja1Wbg8Sn0/A7h7Y331V4cDmVraUreULiTQwSO7N26IxQ3Rg/MQG3szUgP0MWYmjuG0c8zaFB73rnBpZ8xakF/xcRTt2Pb62dkw1VqFhzNc50bN+QvGmtEosIB9z"

},
    "broker-type": "chef"
}

As you can see, the validation file is on one line, as there is no good way to store formatted files in the database, and creating valid JSON with valid new line characters is also not easy. The whole “configuration” block actually gets flattened to a single line in the database, and as such, parsing must occur at some point. I chose the path of least resistance, one that does not require user to create a valid json from the validation file, rather just strip out new line characters.

Let’s take a look at each line in the configuration file:

"name": "openstack_chef_broker"

Is a name of the broker to be used, and it can be pretty much what you like.

"configuration": {

Is the beginning of configuration for the Chef itself.

"install_sh": "http://opscode.com/chef/install.sh",

This is the omnibus installer to be used for installing Chef on your newly installed server. If you have a closed network or have your own installer, it is likely that you will have a custom installer.

"version_string": "11.4.4"

Chef version to be installed.

"chef_server_url": "https://chef.example.com:443",

This is the URL of YOUR Chef server.

"validation_client_name": "chef-validator",

Client name to be used in node’s communication with the Chef server.

"run_list": "role[allinone]",

Initial run list. This is what is going to run after the chef client is installed on the new node.

"environment": "openstack",

Chef environment to be used in the initial Chef run on the new node.

"chef_client": "chef-client",

Binary name for the Chef client installed on the new node. If custom installer installed chef client in a special place or with a special name, specify that here (example: /usr/local/bin/chef-client)

"validation_key":

The validation.pem file. This file must be turned into a single line, and the broker will construct it into it’s original form. Let me know if you have ideas on how to do this better.

"broker-type": "chef"

If you want a Chef broker, you must specify “chef” here. This is to tell razor-server that that’s the broker to use after the node install.

Usage

First, create broker.json file to contain the json:

{
    "name": "openstack_chef_broker",
    "configuration": {
        "install_sh": "http://opscode.com/chef/install.sh",
        "version_string": "11.4.4",
        "chef_server_url": "https://chef.example.com:443",
        "validation_client_name": "chef-validator",
        "run_list": "role[allinone]",
        "environment": "openstack",
        "chef_client": "chef-client",
        "validation_key": "MIIok0h+mSHr1Pbi+v2950H1/7mzd71hXAcRHbmrdytAR+RjgtyibkkZTdSxjLP6o2qEuji0I7DL8cckHG56AvuAKhYoadu+9J/2ahmYr18CSqOmg4Sbh6CPm5edpqaAVTbE3Ec+1wN0IMl8KWtmGCrjpXzH1MDdaLZpIIYqC9DUVgLbd/i61hbiiYlky5wPdIKlKwilRl7alfsGKTUrVq1DuYiIZsDYrnYoLOERbzZU6yUxWSIasfVPJGpT9LvstChFjyjv/73etmhXwwIDAQABAoIBAHH1p1upllVJNMSAhSmLZfTe6Q9nT8unRFH1egcsni8dPXYyVzDQ1ztV3RFNDLjP01ZThqkjoA4TnkHvRnC90yR049eQNxn+7pctcTNW61aAglomMhBcFt7LlcDiiXfD3dVhoIDv4ijpfsNCDvEVain9Dv+krjGVP+dWKLKr2azZbrAnyJKJjKb8DkwFET7UADFgUiCwJCW7RyAAxL40WE1hyYGq03cum5+M+2MGTcCiiAumqIRjYyTioNl73zf/ckBKRBNb6DkVGa1drAwXqMtA0MAaJlQBoJJF9SehSf7rAKVMvQ+esS9kNXZhECgYEA28FKbU3K4mU2khZ8lJgomfelWV48sjeIH3OEuLK7WdKg2LAnNajiLAOHYi0egKiMJPnNyZ8uvOY0JABLHQXv10QKaNlHGqy+aRiRSb1Ya6SlbCg8hMoT29yWGa8k7AclLJSn20/dg0GAEQtegi/k3EEZq0CgYEA148QtVh4ng80d9P81rJEDkraAkKUNJPHBSKFqo1rNwSDsyxxTEXIzZSPYS1mjsdB4IxraiNm3WDhb+EldcSlXvmBDjYJck1WyLCw1JuyAS2VCEtdc38aPWGYZ157decuzkR8CU2TKw48KGSRTtwL8yYqTknqpAmnqf/KkWfVNi8CgYEA1a0AnX+Cwtf/U9UhlarN78fosxj6k5+DcHF2n9DKcvBnDctalMZ+BKX5wfB1NEyu2FU9T8rEO1DragYPA01+hCXYqVJ73OgSzUXiH31IuIID9u/0LyseTWOWoIxWJzdTh44xJ+wJlwNuYXxgjgVGaVZk2niXWTLxtTNEdCz0RpECgYAlholhGIq+8WSv656bfaMtXciAFjkYwhUmhrEAVOgyRr3qpjT/EzL23wLq5u1ws617OtbEPm984I2+XVKZIuerFgJqh+uzE1WlUGUoTgZ6AAZu0DfvkFPwFZpjfGY/y0QxsmhpcjDJkQvV+FP3h4UpCh7ZTDLl5axjgt0v3QSYDwKBgFde/5TO8N8U6lHr1YX+yY8wbQ9sVWPU8qruL5Qx11an1tm9Ja1Wbg8Sn0/A7h7Y331V4cDmVraUreULiTQwSO7N26IxQ3Rg/MQG3szUgP0MWYmjuG0c8zaFB73rnBpZ8xakF/xcRTt2Pb62dkw1VqFhzNc50bN+QvGmtEosIB9z"

},
    "broker-type": "chef"
}

Create broker on the command line:

razor create-broker --json broker.json

Then, create the policy.json file:

{
  "name": "ubuntu_one",
  "repo": { "name": "ubuntu_server" },
  "task": { "name": "ubuntu" },
  "broker": { "name": "openstack_chef_broker" },
  "enabled": true,
  "hostname": "host${id}",
  "root_password": "secret",
  "max_count": "20",
  "rule_number": "107",
  "tags": [{ "name": "ubuntu_small", "rule": ["=", ["num", ["fact", "processorcount"]], 1]}]
}

Create policy:

razor create-policy --json policy.json

Now, your policy is set to use the new Chef broker.
Happy installing!

-eglute

 Posted by at 10:46 pm
Jan 202014
 

If you have not been following google groups and other discussions, this is a summary of the main differences between the original razor and razor-server. The new razor-server is a somewhat different beast than the original razor. If you go to the original git repo, https://github.com/puppetlabs/razor, you will find a note stating “TL;DR: Razor is being rewritten.”

I will not go into the politics of why razor is being rewritten, the google groups provide the official explanation here: https://groups.google.com/forum/#!topic/puppet-razor/q4uCVMmUop0

The main differences are in installation and usage, most things are different, and the new razor-server pretty much carried the name only.

Technology

When it comes to the technologies used, the new razor-server has gotten rid of node.js components and MongoDB. It is now based on JRuby, and, as the name describes, is a server. The server component is TorqueBox, which in turn is based on JBoss. I have not tried it, but I think this will allow razor-server to be more scalable in the future in terms of deployment size. Database used is PostgreSQL, and the different modes now allow for dev/test/prod instances to run on one server.

Microkernel

Razor-server still uses a microkernel, however, the new MK is based on Fedora rather than Tiny Core Linux. Naturally, this makes the new MK much larger.

State Machine

One of the main things in the original razor was the state machine, which would let you know what state your nodes were in at any given time. At the time of writing, there is no easy way to accomplish the same in the razor-server, but there have been discussions about it, so who knows what will happen in the future.

Facts

Original server would collect facts about hardware using lshw. Razor-server does not utilize lshw, and the list of facts at the moment is rather limited. If you need special info about the hardware, you may need additional patches written for the razor-server.

Usage

The original razor ran on command line and as a razor service. Razor-server, on the other hand, runs in a JVM based application server, and all interactions with it are based through it is API. There is no need to curl all commands by hand however, since there is also a razor-client available. Razor-client is a command line tool to interact with the razor-server. The one annoyance with razor-client I have that it is super slow if it is using jruby. This means that you have to manage multiple ruby versions and take care that the client is not using jruby while the razor-server is.

In my opinion, razor-server is a little easier to use when it comes to configuring installers (also known as recipes or tasks) and policies, since everything can be passed as a json file instead of multiple commands on the command line.

Brokers

Razor-server kept the concept of brokers, and currently supports puppet and chef brokers. No broker is still an option. Anybody can also add their own custom broker, however, there is a slight difficulty. Everything in the broker configuration will have to be stored as one entry in the PostgreSQL database. This is a problem if you need to have formatted files transmitted via the broker to the installed server. I had to solve this issue while writing the new Chef broker.

Summary

There is no path of upgrade between the versions of Razor and it is very unlikely that there will be one. These are essentially two different tools, both have great parts and the parts that are so-so.

-eglute

 

 Posted by at 9:44 pm