Ranked Choice Voting

On November 5 Minneapolis Minnesota held elections.  The process of voting could not have been easier.  In less than five minutes I participated in our democracy.  

Finding out who won the various offices up for election was another story.  Our mayoral race had a large number of candidates.  The total number of mayoral candidates was 35 (pulling from the vote information).

It took over two days for the votes to be tabulated by hand using ranked choice voting.  I thought it could be done more quickly. 

And, having a little free time yesterday I wrote up a sample implementation of ranked choice voting.

https://github.com/wcannon/ranked-choice-voting.git

For the curious, the sample implementation program to calculate the election ran the actual mayoral election data in roughly 18 seconds!  

Programs:

election.py 

provides:   ranked-choice voting

input:  csv file containing all votes for an election (1 vote per line, a line consists of 1+ choices for a candidate)

output:  run it and see

e.g.  election.py  election_csv_file.csv

election_votes_generator.py

provides:   a simple way to create a csv of ranked-choice votes

input:  arguments  (OUTPUT_FILE VOTERS CANDIDATES ALLOWED_CHOICES)

e.g.  election_votes_generator.py out.csv  200000  15  3

This would create a simulated set of votes containing 200000 votes, with 15 candidates, where a vote is for 3 candidates in ranked order.

Note:  I did not realize the actual votes had been posted recently.  Therefore, I wrote up this election simulator.  I have added the actual voting information of the mayoral race to the git repo for reference.

Reference links:

http://vote.minneapolismn.gov/results/2013/2013-mayor

http://library.municode.com/HTML/11490/level3/COOR_TIT8.5EL_CH167MUELRUCO.html#COOR_TIT8.5EL_CH167MUELRUCO_167.10AP

http://www.minnpost.com/politics-policy/2013/11/minneapolis-vote-count-process-based-faulty-logic-and-bad-arithmetic

Building custom saltstack modules

Building custom saltstack modules 

This blog post was inspired by Joseph Hall's (Senior Engineer at Salt Stack, Inc.)  presentation about writing custom SaltStack modules.
Thank you Joseph!  http://www.youtube.com/watch?feature=player_detailpage&v=YP73LM8mzL0

SaltStack comes with a large number of extremely useful modules.  

http://docs.saltstack.com/ref/modules/all/

One very handy module is the cmdmod module.
http://docs.saltstack.com/ref/modules/all/salt.modules.cmdmod.html#module-salt.modules.cmdmod

The cmdmod module gives the ability to easily run shell commands against a set of minions.
This provides an ssh like ability across one or more servers in parallel, and is great for ad-hoc commands.

So, why would we need to create a custom SaltStack module?  Eventually, a need arises for new functionality which is not already included in the SaltStack project and does not fit neatly into an ad-hoc shell command.

My example:  Gather metadata about an AWS EC2 instance using a custom SaltStack module.

Background:

AWS metadata is very useful.  A running server configured for a purpose might need to be accessed directly, or identified for use by another service.

Example 1:  To ssh directly to an EC2 instance we need a DNS address or IP address
Example 2:  To add an EC2 instance to a load balancer, we need its AWS instance id.

Amazon recognized this need and created a way for an EC2 instance to retrieve data about itself.  By issuing a specific HTTP GET request from an EC2 instance, the HTTP response returned contains metadata specific to that instance.

AWS reference on metadata:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html

SaltStack to the rescue!  We can run a command on an EC2 instance without knowing its DNS name or IP address, and gather metadata!  This means we do not need to log into the AWS console to get information about an EC2 instance.  And furthermore, we can automate the consumption of such information.

Some metadata in which we might be interested:

• public-hostname (public dns address)
• public-ipv4 (accessible outside aws)
• instance-id (the unique id of an instance)
• placement (availability zone)
• security-groups (names of security groups applied to the instance)

Building the custom module

First:  Choose a module name that does not overlap with the modules supplied by SaltStack.  A name collision would result in our custom module replacing a SaltStack module.  My recommendation is to consistently prepend any module name with either a company name, abbreviation, developer name, initials or etc.  e.g.  wc_ec2_metadata.py

SaltStack reference on custom modules:

http://docs.saltstack.com/ref/modules/

Pre-requisites:

• An AWS account is set up and enabled for EC2
• A salt-minion is installed and running on the EC2 instance

Building our custom module on the EC2 instance:

• Create a directory for building custom saltstack modules (e.g.  # mkdir salt_modules)
• Start writing a module

(e.g. salt-call -m /path/to/my/modules module.fn )


Create a skeleton module and test it works

Test that our module and function will work as a salt module:

• salt-call -m ~/salt_modules  ec2_metadata.test

Expected Output:
local:
    True

Now let's add a useful function with some logging and error handling:
Test that our module and new function will work as a salt module:
salt-call -m ~/salt_modules  ec2_metadata.get_public_dns

Expected Output - a valid DNS name, not exactly what is below:
local:
    ec2-54-221-126-106.compute-1.amazonaws.com

All that is left to do is implement the remaining functions; (public-ipv4, instance-id, placement, security-groups).

Completed module:

There are a number of ways to distribute your new custom SaltStack module to the minions.

On a salt-master you first place your module in the appropriate _modules subdirectory under the "file_roots" location (typically /srv/salt/_modules) and then run one of the following:

• salt \* state.highstate 
• salt \* saltutil.sync_all
• salt \* saltutil.sync_modules  (only updates modules)

Viewing your module and function docstrings is easy using the included sys.doc module.

• salt-call sys.doc my-module  (returns all docstrings in a module)
• salt-call sys.doc my-module.some-func (returning a docstring for just one function in a module)
• Note: these work using 'salt' as well  e.g.  salt myserver sys.doc my-module

My public github repository:

git@github.com:wcannon/saltstack-related.git

https://github.com/wcannon/saltstack-related.git

https://github.com/wcannon/saltstack-related

Community contributed efforts:

Additionally, community contributed efforts (modules, states, grains and more) can be found here.

http://saltstarters.org/

Deploying a typical LAMP application using SaltStack - my experience

Using SaltStack to deploy a typical LAMP application

This article describes my approach to a recent software deployment project.  The goal of the project is to create a secure fast deployment method for multiple LAMP stack applications.  

Previously I have used the python fabric library to deploy software applications.  Having recently set up "instant" software infrastructures using SaltStack I was eager to take advantage of its remote execution capabilities, and node introspection ala grains.

SaltStack Introduction for Deployment

SaltStack is designed to allow parallel remote execution over encrypted channels, and also provides configuration management.  Essentially, SaltStack provides the basis to make things happen quickly, repeatedly, and exactly as desired.    It is also a python based open source project with an active developer community.

Overall Deployment Design

I chose to conceptualize deployment to consist of two distinct pieces.  

Piece # 1:  The assembly or build of the application.  For the three applications I deployed this consisted of combining php files with configuration files.

Details related to Piece # 1:

Each application has a specific github repository.  And, each branch for the repository is deployable if an environment name matches the branch name (convention over configuration).  At build time, the least amount of effort to update the code locally is used (e.g. if a project has already been cloned, and its branch checked out, we simply perform a "git pull")

Piece # 2:  The actions necessary to put the new application into place and be served by the webserver.  This consists of executing a custom python module.

Details related to Piece # 2:

A SaltStack custom module is simply a standard python module that has been synchronized onto the minion.  (e.g.  salt \* saltutil.sync_modules ).  A function in a module can call a function in another module.  This is very handy and is called "cross-calling."  

Note:  If a function in the module is prepended with underscore characters it will not be callable outside the module (iow a 'hidden' function).  

** A third piece exists:  A wrapper that takes user input, and causes piece 1 and piece 2 to happen.  This piece uses docopt for command line argument processing, the salt filesystem, and a salt runner to signal the targeted servers to run their custom python module.

Details related to Piece # 3:

A targeted server is really given a message via the queuing system on the salt-master.  The targeted server identifies it should do something, and then does it, reporting its result back to another queue on the salt-master.  I have used the "grains" system in order to have a server know what roles it has.  (e.g.  is_wepapp1_server: true)

Code Snippets:

DocOpt Sample:

Salt Runner Sample:

SaltStack Custom Module Sample:

Results:

Very fast deployments for a 5 mb application -- (averaged 5.5 seconds)

An application deployment consists of these steps:

1. log onto salt master
2. run the deployment script:   # ./deploy.py  APPNAME ENVIRONMENT                               e.g.  ./deploy.py  webapp1  production
3. observe output of deployment script

Using the same name as above we could call the custom module directly:

# salt -C  'G@environment:production and G@hosting_webapp1:true'  deployment_webapp1.deploy  my-build-file.tar.gz

Positive elements:

• no list of servers to maintain
• no ssh key management 
• scalable, works against lots (read 100's of servers in parallel)
• can use the grains on the servers for other remote ad-hoc execution
• very simple command line usage (dynamically building the module name to publish)
• trivial to add more deployment modules

Negative elements:

• none specific to SaltStack

Items for improvement / enhancement:

• capture deployment history
• rollback capability / specific revision deployment 
• use of a custom returner (rather than stdout on salt-master) -- would prove helpful for queries, or dashboards
• remove older previous deployments (e.g. only retain the last 3)

Choices that worked out well:

• docopt = simply a great library (sorry argparse - my previous favorite)
• keeping a copy of repos / branches on the deployment server = major deployment time reducer
• use of grains on servers (roles & etc) = great way to encourage better management practices, very handy for general usage 
• use of a salt runner rather than calling specific deployment modules via the salt command = allowed me to simplify the usage of the deploy script
• cross-calling salt modules = great way to reuse code, and helps shrink the size of a deployment module

Helpful Links:

docopt:  http://docopt.org/

saltstack:  http://docs.saltstack.com/

saltstack installation:  http://docs.saltstack.com/topics/installation/index.html

saltstack targeting:  http://docs.saltstack.com/topics/targeting/grains.html

saltstack custom module:  http://docs.saltstack.com/ref/modules/index.html
saltstack runner: http://docs.saltstack.com/ref/clients/index.html#runnerclient

saltstack module cross calling:  http://docs.saltstack.com/ref/modules/index.html?highlight=cross#cross-calling-modules