Things to know before you configure

Collins configuration files are type safe. That is, you can't create a configuration that will lead to a runtime error. If collins detects a configuration error it will fail to start. If you modify a config after the system starts and that config is invalid, it will not load and collins will complain loudly. If this is not the case, please file a bug.

Every type of configuration comes with a reference config. Reference configuration files can be found in the github repo. Reference files are not modified by users, they serve to provide Collins with default values as well as to tell it what the appropriate value types are (lists, strings, etc). Most reference configurations come with some level of documentation as well. In general you should use and make modifications to conf/application.conf.

The configuration file that you create for collins is overriding the values found in the reference configs. There is a sample configuration in the github repo for you to use as a starting point.

You can pass your configuration file to collins by using a system property. Pass -Dconfig.file=/path/to/file will cause that file to be used for the application.

Configuration options that don't fit anywhere else

Application Specific

These configuration aspects are specific to the play framework.

application.secret="AbFgHx0eJx8lalkja812389uasdlkajsdlka98012398uasdlkasdklajsd81298"
parsers.text.maxLength=1048576
evolutionplugin = disabled

Concurrency

These configuration aspects manage various concurrency components of the system. This will dictate how much work Collins can do at any given time.

concurrency {
  actorCount = 48
  timeout = 10 seconds
}

Crypto

The only crypto configuration available is the key. The crypto key must not be changed after the system is initialized for the first time or you will be unable to decrypt data.

crypto.key = "lkkajsdlkajsdlkajsdlkajsdlkajsdlkajsdlkajsdlkajsdlkajsdlkajsdla"

Database

db {
  collins {
    logStatements = false
    user = "sa"
    password = ""
    driver = "org.h2.Driver"
    url = "jdbc:h2:mem:play;IGNOERCASE=true"
  }
}

db {
  collins {
    logStatements = false
    user = "root"
    password = ""
    driver = "com.mysql.jdbc.Driver"
    url = "jdbc:mysql://localhost/collins?autoReconnect=true&interactiveClient=true"
  }
}

LSHW Parsing

Due to how LSHW handles PCIe flash devices, you have to give the parser a hint as to which attribute values indicate that the device is in fact a flash storage device. Here you can specify the string to match on and the size to use if the match is found.

lshw {
  defaultNicCapacity = 10000000000
  flashProducts = ["flashmax","fusionIO","TachION Drive"]
  flashSize = 1400000000000
}

Query Log

The query log is primary used for development but allows you to turn on database query logging and API (find/search) logging.

querylog {
  enabled = false
  includeResults = false
  prefix = "QUERY: "
  frontendLogging = false
}

Setting up user authentication and authorization

Authentication Namespace authentication

Collins supports a variety of authentication backends along with a flexible API endpoint based permissions system. Some features also have specific permissions.

Collins comes bundled with a small in memory authentication mechanism for testing, file based authentication, and LDAP based authentication. There is not yet support for LDAPS authentication so if you use LDAP we recommend running a slave on the collins host.

The permissions example below creates two user groups, admins and tools. User groups are just named groups of users that are stored in the YAML file. This is useful for people authenticating with flat files or if you want to store some group information here. The admins user group has two users, alan and bmatheny. The tools user group has three users. There are two feature permissions specified, one for rate limiting and the other for who can see passwords. The last permission is for the powerStatus API endpoint and is open to a broader group of users. Every API permission name is documented in the API documentation.

users:
  admins:
    - "alan"
    - "bmatheny"
  tools:
    - "jetpants"
    - "itouch"
    - "phil"
permissions:
  feature.noRateLimit:
    - "u=tools"
  feature.canSeePasswords:
    - "u=admins"
    - "u=tools"
    - "g=infra"
  controllers.AssetManagementApi.powerStatus:
    - "g=infra"
    - "g=ops"
    - "u=admins"
    - "u=tools"

authentication {
  cacheCredentials = false    
  cacheTimeout = 15 seconds
  cachePermissionsTimeout = 30 seconds
  type = default
  permissionsFile = "conf/permissions.yaml"
  adminGroup = [Infra]
}

LDAP Authentication Namespace authentication.ldap

At Tumblr LDAP is the primary authentication mechanism. We have tested that this code works with at least RFC 2307, IPA, and the OS X admin server.

authentication {
  type = ldap
  ldap {
    host = "localhost"
    groupAttribute = "uniqueMember"
    searchbase = "dc=example,dc=org"
    usersub = "ou=people"
    groupsub = "ou=groups"
  }
}

File Authentication Namespace authentication.file

The file format used is a variation on that used by Apache. You can find a sample script for generating users and passwords in scripts/gen_passwords.sh.

authentication {
  type = file
  file {
    userfile = "conf/users.conf"
  }
}

Making Collins faster

cache {
  class = "collins.cache.GuavaCache"
  timeout = 10 minutes
}

Hooking into lifecycle events in collins

Namespace callbacks

Collins supports a callback mechanism that allows you to trigger events when the configured callbacks are matched. A short example is probably most useful here.

callbacks.registry {
  hardwareProblem {
    on = "asset_update"
    when {
      previous.state = "!isMaintenance"
      current.state = "isMaintenance"
      current.states = [ HARDWARE_PROBLEM ]
    }
    action {
      type = exec
      command = "/sbin/notify <tag> sysadmin@example.org"
    }
  }
}

In the above configuration we have registered a callback named hardwareProblem. This callback will be triggered when an asset is moved to have a status of Maintenance with a state of HARDWARE_PROBLEM. When the callback is triggered we will execute a notify command that will email a systems administrator.

There are more useful examples in the recipes documentation.

A callback is triggered by an event which is specified via the on config value. The value is then matched using a matcher which is specified via the when config value. If the matcher succeeds the specified action is taken.

callbacks {
  enabled = true
  class = "collins.callbacks.CallbackManagerPlugin"
  registry {
  }
}

Registry Namespace callbacks.registry

The registry holds all your configured callbacks.

Collins feature flags

Namespace features

There are a number of parts of collins that you may wish to customize based on your environment. These are referred to in the reference configuration and elsewhere as feature flags. Available feature flags are documented below.

features {
  allowTagUpdates = []

  allowedServerUpdateStatuses = []

  defaultLogType = Informational

  deleteIpmiOnDecommission = true

  deleteIpAddressOnDecommission = true

  deleteMetaOnDecommission = false

  deleteSomeMetaOnRepurpose = []

  encryptedTags = []

  ignoreDangerousCommands = []

  intakeSupported = true

  hideMeta = []

  noLogAssets = []

  noLogPurges = []

  sloppyStatus = true

  sloppyTags = false
}

features {
  allowedServerUpdateStatuses = [ PROVISIONED, PROVISIONING ]

  encryptedTags = [SYSTEM_PASSWORD, LOCATION]

  noLogAssets = [chatty_config, other_chatty_config]

  noLogPurges = [SYSTEM_PASSWORD]
}

Display trending data

Overview Namespace graph

• Collins has an easy to implement plugin for graph and trending systems. The graph plugin will display embedded graphs or charts for assets, or just links to those systems (depending on the graph plugin). Once enabled, you will find an additional tab on the asset details screen named 'Graphs' with all the configured information available in it.

  There is currently support for Fibr, the OpenTSDB visualization tool we built at Tumblr and ganglia. While there are plans to open source fibr in the near future, it is likely an early November release.

  Because graph plugins are easy to implement, we hope that the community will contribute plugins for other common similar systems such as graphite, or munin.

  If you are interested in building and contributing a plugin and have any questions, please feel free to reach out on the mailing list.

• 

graph {
  class = "collins.graphs.FibrGraphs"
  enabled = false
  queryCacheSize = 2000
  queryCacheTimeout = 60 seconds

  FibrGraphs {
    annotations = [deploy]
    customMetrics = {}
    defaultMetrics = [
      SYS/LOAD,
      SYS/MEM,
      SYS/NET,
      SYS/PROC,
      SYS/IO-UTIL
    ]
    timeframe = 1 hour
    url = "http://fibr.d2.tumblr.net"
}

Fibr Graphs Namespace graph.FibrGraphs

At Tumblr, Fibr is our primary trending, graphing, and systems visualization tool. There is more information on fibr in the tool document.

graph.FibrGraphs {
  customMetrics = {
    mysqlHosts {
      selector = "PRIMARY_ROLE = \"DATABASE\""
      metrics = [
        MYSQL/COMMANDS,
        MYSQL/SLOW,
        MYSQL/THREADS,
        MYSQL/NET
      ]
    }
    memcacheHosts {
      selector = "POOL = MEMCACHE*"
      metrics = [
        MEMCACHE/COMMANDS,
        MEMCACHE/EVICTS,
        MEMCACHE/ITEMS,
        MEMCACHE/HITRATIO
      ]
    }
  }
}

In the above example, hosts that have a PRIMARY_ROLE of DATABASE will display in addition to the default metrics, the MySQL specific metrics. Assets with a POOL matching MEMCACHE* will additionally display memcache specific metrics.

Ganglia Graphs Namespace graph.GangliaGraphs

• Ganglia is a distributed metrics generation, collection, and visualization system. Collins integration relies on recent ad-hoc-views support.

• 

graph {
  class = "collins.graphs.GangliaGraphs"
  enabled = true
  GangliaGraphs {
    url = "http://my-ganglia.domain.local/"
 }
}

IPAM Configuration

Namespace ipAddresses

The Collins IPAM configuration is pretty straight forward. You specify more or less a set of pools that are available for allocation. Once configured, assets can have addresses from any of these pools assigned to them.

ipAddresses {
  pools {
    DEV {
      startAddress = "172.16.4.4"
      network = "172.16.4.0/28"
    }
    ROW_1_1 {
      name = "ROW_1.1"
      startAddress = "172.16.64.5"
      network = "172.16.64.0/24"
    }
  }
}

The above configuration now allows you to allocate addresses out of two pools, a DEV pool and a ROW_1.1 pool. Notice that the ROW_1.1 pool had to specify a name. This is because an object name can not contain a period so must actually set the name manually as a string. The DEV pool did not have to do this. At Tumblr we have found pool names most useful for mapping to a particular VLAN and handling VLAN assignments using this information.

Pool Config

Each pool can take a name, gateway, startAddress and network.

Configuring your IPMI network and authentication setup

Namespace ipmi

The IPMI configuration is used to drive generation of IPMI addresses, usernames and passwords. This information is used for power management operations such as power cycling an asset or lighting up an asset during the intake process.

ipmi {

  passwordLength = 12
  randomUsername = false
  username = "root"
  network = "10.0.0.0/16"
  startAddress = "10.0.0.3"

}

Monitoring Collins

Namespace metricsreporter

Monitoring Collins using Coda Hale's Metrics and Chris Burrough's metrics-reporter-config project is relatively straightforward. See Metrics' documentation for how to report specific metrics, and the metrics-reporter-config documentation for how the configuration file should be formatted.

metricsreporter {
  
  enabled= false
  configFile = "conf/metrics-reporter-config.yaml"

}

For when a single data center just isn't enough

Namespace multicollins

When Tumblr grew into a second datacenter, we didn't want to bother with having to setup multi-master replication across sites or do some kind of master site replication scheme. Instead what we decided is that we would make each Collins instance aware of each other, and allow them to query and interact with each other. Don't worry, no crazy zookeeper magic going on here.

If you have multiple independent collins installations setup, to make them aware of each other you have to do two things.

1. Configure multi-collins
2. Create an appropriate asset in each collins instance

multicollins {
  enabled = false
  instanceAssetType = DATA_CENTER
  locationAttribute = LOCATION
  thisInstance = NONE
}

Once you have configured multicollins you will now be able to use the remoteLookup API parameter for asset finds. The Ruby client automatically handles fetching remote assets when needed. Additionally, you should find a 'Remote Search' checkbox in the search UI.

Find similar assets

Namespace nodeclassifier

A pretty common thing to want to be able to do is ask collins to tell you what other assets are like another. At Tumblr we use this for bulk provisioning, where I might want to be able to do something like say 'provisioning me 100 servers that are just like this one'. This is the problem that is solved with the node classifier. A node classifier describes a node class (e.g. web, queue, database, etc) and allows you to query for assets described by the classifier.

Much like with the MultiCollins configuration, we reuse assets to configure our node classifiers. In order to create a node classifier create a new asset (we prefer Configuration assets), and set tags on it that describe the traits of similar assets. For instance you might create a Configuration asset named web_classifier and set the NIC_SPEED to 10000000000, the MEMORY_SIZE_TOTAL to 34359738368, and the TOTAL_DISK_STORAGE to 500107862016. This configuration now describes a web class host with a 1Gb NIC, 32GB of RAM, and a 500GB disk. Be sure to set an IS_NODECLASS tag to true on the asset.

With the above configuration specified you can now use the find similar API endpoint. You also in the web UI should find links for finding similar assets.

nodeclassifier {
  assetType = CONFIGURATION
  identifyingMetaTag = IS_NODECLASS
  displayNameAttribute = NAME
  excludeMetaTags = [IS_NODECLASS, NOTE]
  sortKeys = []
}

Power requirements for assets

Namespace powerconfiguration

Overview

Depending on the environment, each asset may have a different power configuration. A power configuration specifies the number of power units required per asset, the number of power components required per power unit, and various requirements for each power component.

Definitions

• An asset may have an associated power configuration
• A power configuration consists of one or more power units
• A power unit consits of one or more power components (e.g. power strip, power outlet, etc)
• A set of power units represents a complete power configuration

You can think of an asset power configuration as looking something like the following.

• PowerConfiguration
  • Power Unit 1
    • Power Component 1
    • Power Component 2
    • Power Component n
  • Power Unit 2
  • Power Unit n

Configuration

Note that this configuration heavily influences the intake process as well as what API parameters are available for asset updates. You can get a sample for what the asset looks like in the Asset Get JSON response.

Messages

It is possible to customize how the various components are labeled. The default bundled messages defines the following messages.

• powerconfiguration.unit.port.label - Power Strip {0}
• powerconfiguration.unit.outlet.label - Outlet {0}

Notice that the configured components each get a unit label. This allows you to customize how the intake screen looks.

With great power, comes great responsibility

Namespace powermanagement

Assuming you would like the ability to perform power management operations on devices, such as power cycling them or powering them off, you will want to provide a power management configuration.

powermanagement {
  enabled = true
  command_template = "ipmitool -H <host> -U <username> -P <password> -I lan -L OPERATOR"

  commands {
    powerOff = ${powermanagement.command_template}" chassis power off"
    powerOn = ${powermanagement.command_template}" chassis power on"
    powerSoft = ${powermanagement.command_template}" chassis power soft"
    powerState = ${powermanagement.command_template}" chassis power status"
    rebootHard = ${powermanagement.command_template}" chassis power cycle"
    rebootSoft = ${powermanagement.command_template}" chassis power reset"
    identify = ${powermanagement.command_template}" chassis identify <interval>"
    verify = "ping -c 3 <host>"
  }
}

Note A command must be specified for every power action.

Setup provisioning for your environment

Namespace provisioner

The provisioner plugin adds asset provisioning capabilities to collins which can be driven via the API or the web UI.

provisioner {

  allowedStatus = [Unallocated, Maintenance, Provisioning, Provisioned]

  allowedType = [SERVER_NODE, SERVER_CHASSIS]

  enabled = false

  cacheTimeout = 30 seconds

  checkCommand = "/bin/true"

  command = "/bin/false"

  profiles = "test/resources/profiles.yaml"

  rate = "1/10 seconds"
}

Provisioning Rates

Rates are expressed in terms of the number of successful requests per time period that a user can make. Specified as #/duration where # is an integer and duration is the duration. Examples:

• 1/30 seconds - once per 30 seconds
• 5/1 minute - five times per minute (same as 1/20 seconds)
• 1/0 seconds - no rate limit
• 0/0 seconds - disallowed

Substitutions in Commands

A few substitutions are available in the command and checkCommand

• <tag> - the asset tag being provisioned
• <profile-id> - the id of the profile being used for provisioning
• <notify> - the contact information supplied by the user doing the provisioning
• <suffix> - any suffix to append to the hostname as provided by the user (if allowed)
• <logfile> - a logfile to use, defaults to /tmp/<tag>-<profile-id>.log

Provisioner Profile

The provisioner profiles file is what tells collins what types of assets can be provisioned. It is a fairly simple format and leaves most actual work up to the command and check command configured CLI tools. At Tumblr we have implemented a variety of simple CLI tools to handle a variety of provisioning scenarios. Our infrastructure automation framework is called visioner, some sample visions will be distributed so you can see how some of our provisioning works.

The keys in the reference file below (databasenode, devnode, webnode) are automatically set to be the NODECLASS tag of the asset in collins. This key is also reference above as <profile-id> and is how the provision process knows what configuration to use. A profile can have any number of key/values associated with them of any type, but has a few reserved key/value pairs.

profiles:
  databasenode:
    label: "Database Server"
    prefix: "db"
    primary_role: "DATABASE"
    requires_pool: false
    contact: dbteam@company.com
    contact_notes: http://wiki.company.com/servers/databasenode
    clear_attributes:
    - TEST_DATABASE
    - DONT_WANT_THIS_ATTRIBUTE_EITHER
  devnode:
    label: "Dev Machine"
    prefix: "dev"
    allow_suffix: true
    primary_role: "DEVEL"
    pool: "DEVEL"
    attributes:
      DEV_MODE: true
  hadoopnode:
    label: "Hadoop Server"
    prefix: "hadoop"
    allowed_classes:
    - storage_class
    - service_class
  webnode:
    label: "Web Server"
    prefix: "web"
    requires_secondary_role: true

Working with SoftLayer

Namespace softlayer

Although Collins was initially developed as Tumblr built out its own datacenter, we found it an excellent replacement for the SoftLayer web interface. The SoftLayer plugin provides support for cancelling servers, activating servers from the spare pool, and decorating tags that are SoftLayer specific.

Note The Collins system assumes that asset tags associated with SoftLayer assets are of the form sl-ID where ID is the SoftLayer ID.

softlayer {
  enabled = true
  username = "jimbob123"
  password = "lkajsdlkjaas09823lkajdslkajsd"
  allowedCancelStatus = [Unallocated, Allocated, Maintenance]
}

The above configuration would perform operations as user jimbob and allow you to cancel assets in the specified status.

Better search

Namespace solr

If the Solr plugin is enabled, Collins will use Solr to index assets. Since you can run Solr either embedded (bundled into Collins) or using an external instance, you should almost definitely keep this enabled.

Once the Solr plugin is enabled you will notice a few new features pop up in Collins. First, you can now query collins using the collins query language. This allows you to perform more advanced queries using ranges (how many assets have between 4 and 8 CPUs, give me assets with more than 2TB of usable storage) and much more sophisticated boolean operators. You can also sort results by any tag which is nice. You can find example queries in the collins query language section of the recipes guide.

Taking sexy back

STEVE TODO