Infrastructure management for engineers
Collins exists to drive infrastructure automation.
Because of the loosely coupled design of Collins, consistently applied conventions are a system requirement. This document serves as a guide to those conventions as well as the basic core concepts of the collins system. If you're just interested in the basic howto or screenshots, click here.
Shopify has adopted Collins, and implemented some missing parts of the stack. It is part of the "Genesis Stack". This site is a fork of Tumblr's documentation, as it applies to Shopify. Eventually, we'd like contribute our changes back to Tumblr's Shopify, but maintain this fork while we develop and refine the features we are adding. The original upstream documentation is available at Tumblr's Github.io page
Collins consists of the core application, a set of client libraries, the collins-shell, and an automation toolchain. Nothing but collins core is required.
Collins provides an easy to use API on top of a simple data model. You can extend collins through callbacks, plugins, and light-weight collins scripts.
Collins is extremely dumb. It knows about assets, their meta-data and asset logs. You can think of collins as a key/value store where each asset has its own set of key/value pairs. There are no relationships between assets other than the ones you, through convention, derive. The API makes it trivial to create and manage the tags (meta-data, key/value pairs) associated with an asset, and to query based on those tags.
Collins is intentionally dumb. It worries about basic authentication, clean API interactions, and data persistence. If you start thinking, "Hey, I should build X into Collins", you probably shouldn't. Collins supports both a plugins architecture (for things that actually in some way change the behavior of collins) as well as a very usable API (including clients in Python, Ruby and Bash). Nearly everything you might want to do can be accomplished via the API and anything that can't is doable as a plugin.
By infrastructure engineers, for infrastructure engineers.
We looked at a dozen or so open source and commercial products before coding began. Having an API and flexible workflow were requirements. The intake support for our specific hardware intake process also needed to be accounted for. We also needed something that would support not just virtualized systems, but was more tailored towards physical hardware. We found nothing that met the requirements we had. Things we looked at included:
Some of these are strictly for IPAM, some handled parts of what collins does. None of these were good fits for the initial problems we were trying to solve.
Collins can sit on top of existing asset management systems (we have done this at Tumblr with both SoftLayer and Amazon) or be the infrastructure source of truth for your entire environment. I personally found openstack to be the closest fit for what we were looking for but the number of changes required to make it work for Tumblr were substantial.
Up and running in 5 minutes.
Collins has 1 software requirement, Java 1.6 (with the unlimited strength JCE).
Collins is developed using Java 1.6 and in the tests I did with Java 1.7 on OS X collins performed poorly. Typing java -version
should display something like java version "1.6.0_35"
If you would like your data to persist between restarts, you should also have MySQL 5.5 (5.1 may work but is untested).
scripts/setup and follow the instructionsNote that the above method will only work with MySQL. If you don't have MySQL available you can follow the instructions below which will use an in memory database.
$HOME/src/play-2.0.4cd ~/src/ & & git clone git@github.com:Shopify/collins.git~/src/collins type ~/src/play-2.0.4/play and hit enter (this will take a minute the first time)compile and hit enter, you should see no errors. We're just testing your environment, you normally won't do this.run and then visit http://localhost:9000 in your browser. To run with a different configuration file (the default is conf/application.conf), instead of typing run type run -Dconfig.file=/path/to/alt.conf. Login with username collins and password collins.There is a quickstart screencast up here. This has changed slightly since it was published, but essentially just walks you through the above an additionally installs and sets up the collins-shell.
There is a supplied population script at scripts/populate.sh which will create 10 assets in your local collins instance for you to play around with.
There is a supplied package script at scripts/package.sh which will produce a fat zip with all the requirements and dependencies. This is what we deploy to production at Tumblr.
There is a supplied init script at scripts/collins.sh. This is a useful starting point for running the app and is what we use to run the app in production at Tumblr. This script is also bundled by the scripts/package.sh script. Note that the supplied init script requires daemon.
If you want to help develop collins, you'll need to follow the instructions in "Software Setup" above. Then, to use your favorite IDE, follow the instructions on how to set it up using play. Setting up your preferred IDE with Play Framework
Quick links to useful resources
A number of people were responsible for Collins growing into the useful tool that it has.