The Collins tool chain is at your disposal

The Collins tool chain consists of a suite of tools, applications, and libraries you can use along with your collins installation. You technically don't have to use any of these, but hopefully as you'll see in the recipes guide, they're pretty damn useful.

These tools all share the same licensing as Collins core (Apache 2) and include only free and open source software. Most code tends to be Ruby based but when appropriate we have tried to either provide an SDK for other languages (as well as bash) or ensure that the code is user servicable. More details about these tools can be found throughout the rest of this document.

API access from the language of your choice

We realize that you may want to access Collins from a variety of contexts and as such we provide an SDK for the languages and environments we use at Tumblr. While we don't have an SDK available for every language, the API is well documented so anyone inclined should have no problem implementing one.

The reference implementation is implemented with Ruby, and is considered the most complete SDK. Anyone looking for a sample implementation is encouraged to review the Ruby collins-client code. Anytime a new API endpoint is implemented, in addition to documentation, we include support in the Ruby collins client and if applicable in the collins shell.

The ruby gem is available via rubygems.org/gems/collins_client.

In addition to the Ruby implementation we have a modest Bash library available that has support for much of the basic core API endpoints. This is the library we use for Invisible Touch.

We have a very basic python client available as well but it pretty much only implements the API code needed to support deploys at Tumblr. We would welcome a community contribution.

CLI access to Collins and your infrastructure

Overview

One of the tools in the collins tool chain is the collins shell. The collins shell provides a command line tool with complete API access to all API endpoints. The collins shell also provides an interactive console, built around pry giving you a full ruby environment for task automation.

Installation

Note that rvm is not a requirement for using the collins shell. Ruby 1.9.x is however. I recommend using ruby 1.9.3 because that's what we develop with at Tumblr. 1.9.2 probably also works, 1.8.x will not work. There is additional information on installing rvm here.

$ curl -L get.rvm.io | bash -s stable --ruby
$ rvm install 1.9.3
$ rvm use 1.9.3
$ gem install collins_shell

Setup

By default collins-shell looks for a configuration file in ~/.collins.yaml. You can specify an alternate location to use with the --config=/path/to/collins.yaml option or using the COLLINS=/path/to/collins.yaml environment variable. There is a sample config file below.

---
host: https://collins.example.org
username: "joeuser"
password: "secret"
timeout: 30

If configuration files aren't your thing you can just all the variables from the config in via command options. You can specify the host, username and password via the --host, --username, and --password options. If you just specify --password (without a value), or your configs are missing a password, you will be prompted for one. The password prompt masks your input.

Usage

Before using any command, check out the help. You can see help for a command by running:

collins-shell help <command>

collins-shell <sub> help <command>

<sub> is something like asset, ipmi, or ip_address and <command> is something like create, delete or get.

> collins-shell
Tasks:
  collins-shell asset <command>                         # Asset related commands
  collins-shell console                                 # drop into the interactive collins shell
  collins-shell help [TASK]                             # Describe available tasks or one specific task
  collins-shell ip_address <command>                    # IP address related commands
  collins-shell ipmi <command>                          # IPMI related commands
  collins-shell latest                                  # check if there is a newer version of collins-shell
  collins-shell log MESSAGE                             # log a message on an asset
  collins-shell logs TAG                                # fetch logs for an asset specified by its tag. Use "all" for all logs
  collins-shell power ACTION --reason=REASON --tag=TAG  # perform power action (off, on, rebootSoft, rebootHard, etc) on an asset
  collins-shell power_status                            # check power status on an asset
  collins-shell provision <command>                     # Provisioning related commands
  collins-shell state <command>                         # State management related commands - use with care
  collins-shell tag <command>                           # Tag related commands
  collins-shell version                                 # current version of collins-shell

> collins-shell asset
Tasks:
  collins-shell asset create --tag=TAG                  # create an asset in collins
  collins-shell asset delete --reason=REASON --tag=TAG  # delete an asset in collins (must be cancelled)
  collins-shell asset delete_attribute KEY              # delete an attribute in collins
  collins-shell asset find --selector=key value         # find assets using the specified selector
  collins-shell asset find_similar TAG                  # get similar unallocated assets for a gven asset
  collins-shell asset get TAG                           # get an asset and display its attributes
  collins-shell asset help [COMMAND]                    # Describe subcommands or one specific subcommand
  collins-shell asset set_attribute KEY VALUE           # set an attribute in collins
  collins-shell asset set_status --reason=REASON        # set status, state, or both on an asset


> collins-shell asset help get
Usage:
  collins-shell asset get TAG

Options:
  [--config=CONFIG]      # YAML configuration file
  [--debug]              # Debug output
  [--host=HOST]          # Collins host (e.g. http://host:port)
  [--password=PASSWORD]  # Collins password
  [--quiet]              # Be quiet when appropriate
  [--timeout=N]          # Collins client timeout
                         # Default: 30
  [--username=USERNAME]  # Collins username
  [--confirm]            # Require exec confirmation. Defaults to true
                         # Default: true
  [--exec=EXEC]          # Execute a command using the data from this asset. Use {{hostname}}, {{ipmi.password}}, etc for substitution
  [--logs]               # Also display asset logs
  [--remote=REMOTE]      # Remote location to search. This is a tag in collins corresponding to the datacenter asset

get an asset and display its attributes

• 

  Collins Get

  The collins shell provides you with all the same information available to you in the web UI, in an easy to digest format.

Universal Options

The following list of options can be specified with every collins shell command. In some cases (development) it may be useful to set the environment variable COLLINS_DEBUG=true which will turn on HTTP level debugging. Use this in conjunction with the --debug option.

--config=CONFIG       # YAML configuration file
--debug               # Debug output
--host=HOST           # Collins host (e.g. http://host:port)
--quiet               # Mostly used in conjunction with commands that have an --exec option
--timeout=SECONDS     # Seconds to allow for operation, defaults to 30
--username=USERNAME   # Collins username
--password=PASSWORD   # Collins password

Common Options

Many options will allow you to specify either a selector (matching many assets) or a tag (matching a single asset). A tag is the asset tag, a selector is a space separated list of key:value pairs that are asset keys and values.

# Allocated web servers (note that multiple selectors are separated by a space)
--selector=hostname:'^web.*' status:Allocated
# Allocated master database servers in the main pool
--selector=primary_role:database pool:main secondary_role:master status:Allocated
# Asset with tag 001923
--tag=001923

Tips and Tricks

There are useful examples in the recipes guide.

The Console

The collins shell console provides an interactive environment for working with your assets. When you're inside the collins console, you're working in a Ruby environment so you have complete access to all the language features of Ruby.

Start a console session

> collins-shell console

Change your context to a PRIMARY_ROLE. Since PRIMARY_ROLE is a tag, and the console knows this, you will get context sensitive information about primary roles

collins / > cd /PRIMARY_ROLE

Now that you're in the PRIMARY_ROLE context, use ls to see what tag values are available.

collins /PRIMARY_ROLE > ls

We're interested in assets with a PRIMARY_ROLE of DEVEL, so let's change context again.

collins /PRIMARY_ROLE > cd DEVEL

Now let's again use ls to list items in this context, but since we know these are assets, let's also format them so they include the hostname and status and only show assets formatted like this that have the phrase 'collins'.

collins /PRIMARY_ROLE/DEVEL > ls --format='{{hostname}} {{status}} {{tag}}' --grep=collins

We have found the asset we're interested to it, let's change context again

collins /PRIMARY_ROLE/DEVEL > cd sl-91016

Now that we're finally in an asset context, the prompt tells us this by ending the prompt with an asterisk. Once you're in an asset context you have a number of new commands and methods available to you.

collins /PRIMARY_ROLE/DEVEL/sl-91016* > ls
collins /PRIMARY_ROLE/DEVEL/sl-91016* > cat -b
collins /PRIMARY_ROLE/DEVEL/sl-91016* > cat /var/log/messages
collins /PRIMARY_ROLE/DEVEL/sl-91016* > cat /var/log/NOTE
collins /PRIMARY_ROLE/DEVEL/sl-91016* > asset.created.to_s
collins /PRIMARY_ROLE/DEVEL/sl-91016* > power?
collins /PRIMARY_ROLE/DEVEL/sl-91016* > reboot!

Context paths are relative, like in a regular shell. Let's check out another asset and get its status

collins /PRIMARY_ROLE/DEVEL/sl-91016* > cd ../sl-102313
collins /PRIMARY_ROLE/DEVEL/sl-102313* > stat

Generate kickstart and iPXE configs from Collins

This is replaced in the Genesis stack by Alchemy Transmuter

Physical intake, burn-in and rescue image

This is replaced in the Genesis stack by Alchemy Linux

Content

Infrastructure automation framework in Ruby

This is replaced in the Genesis stack by Alchemy Transmuter Spells

Content

Infrastructure visualization with OpenTSDB

Content