In Moving to Librarian-puppet I mentioned that we needed to keep 3 different sets of puppet stacks around. I thought I’d expand on it.

There will likely be a part 2, & maybe 3, as I describe what we have inside our puppet provisioner code for packer, and then later for docker…

We’re using 2 different sets of puppet stacks, similar, but not quite the same, and testing a third:

1. one to build AMI’s, installing software, but typically not starting it.
2. the full build
3. testing docker

These are different because we install and start different parts based on which portion of the build is involved.

Packer

For Packer, we want to install all the pieces, especially packages with large dependency chains. We don’t want to start services, but putting config files in place is helpful. We also currently put a copy of the code git checkout in place, so that VCSrepo updates can happen very quickly, only needing to download delta’s from upstream.

Full Build

In the full build, we need to make sure that everything is installed, and running. We also add some runtime cache cleanup code that wouldn’t be helpful in the packer AMI build, and also leave out some scripts that update mcollective caches and drop extra facts into it’s cache. Extra Yaml in mco

Docker

Docker is new target for us. We’re hoping to move to move to Docker for building containers with our code + virtualenv & running either celery or uwsgi. We hope to be able to use the same artifacts from dev-> prod, and that it will make life easier to update running versions of code.

However, we will still build AMI’s for the host systems, running mcollective & puppet, sensu for monitoring, and probably the nginx webserver/proxy for the uwsgi containers. We’ll also include in the packer built AMI’s the docker images, so these systems can be deployed via autoscaling.

This will mean we’ll have a different set of requirements for puppet in the docker systems. We’ll be getting rid of much of the code deployment related functions, but still keeping most of the other parts. And we’ll have to add docker related functionality & mgmt.

So… The office hit a point where I had way too many modules installed from the forge & git, and it was past the point of being a problem.

(Beyond the maintenance problem, we also hit an issue for using Packer & Docker )

We’ve implemented a hybrid roles & providers model, so we end up w/ quite a few custom modules for our needs.

We had 45 modules. It was starting to get difficult to find our library modules, from our custom ones.

I’ve seen people talking about Librarian Puppet, and about R10K.

(FYI: http://garylarizza.com/blog/2014/02/18/puppet-workflow-part-3/ is a very explanation of why you might use R10K, and how..)

However, I really didn’t want to have to move all my code to a new repo, and especially didn’t want to move each module to it’s own repo. This is all being maintained by one person right now, and the dev’s have enough trouble w/ it being in it’s own repo, vs the main code repo.

Eventually I realized that I could move the custom modules to their own module directory, and use the main “modules” directory for librarian-puppet.

In order for this to work, I had to modify puppet.conf to have

modulepath =  $confdir/modules-local:$confdir/modules

On my server w/ (older, pre 3.5) dynamic environments, I used:

    modulepath =  $confdir/environments/$environment/modules-local:$confdir/environments/$environment/modules

Once the Puppetfile was created, I just had to run:

librarian-puppet install

Here’s the Puppetfile we used, and what it produces:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

forge "http://forge.puppetlabs.com"

mod "rodjek/logrotate"

mod "elasticsearch/logstash"
mod "elasticsearch/elasticsearch"
mod "sensu/sensu"
mod "garethr/graphite"

mod "puppetlabs/ntp"
mod "puppetlabs/mcollective",
  :git => 'https://github.com/matthewbarr/puppetlabs-mcollective.git',
  :ref => 'kensho'

mod "puppetlabs/vcsrepo"
mod "puppetlabs/git"


mod "jfryman/nginx"
mod "puppetlabs/inifile"

Generates:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

modules
├── elasticsearch-elasticsearch (v0.3.2)
├── elasticsearch-logstash (v0.5.1)
├── footballradar-python (v0.1.0)
├── garethr-graphite (v0.3.0)
├── ispavailability-file_concat (v0.1.0)
├── jfryman-nginx (v0.0.9)
├── mcollective (???)
├── nanliu-staging (v0.4.0)
├── puppetlabs-activemq (v0.2.0)
├── puppetlabs-apache (v0.6.0)
├── puppetlabs-apt (v1.4.2)
├── puppetlabs-concat (v1.1.0)
├── puppetlabs-firewall (v1.1.1)
├── puppetlabs-git (v0.0.3)
├── puppetlabs-inifile (v1.0.3)
├── puppetlabs-java (v1.1.1)
├── puppetlabs-java_ks (v1.2.3)
├── puppetlabs-ntp (v3.0.3)
├── puppetlabs-rabbitmq (v4.0.0)
├── puppetlabs-stdlib (v4.2.1)
├── puppetlabs-vcsrepo (v0.2.0)
├── richardc-datacat (v0.4.3)
├── rodjek-logrotate (v1.0.2)
└── sensu-sensu (v1.0.0)

Which is 26/45 of my original modules. I was also able to remove a few dependencies, and upgraded a few of the modules. (Which why you only see 24 modules listed above..)

It also meant that I had to extract my changes to a few upstream modules, and send them up as PR’s. In the end, the only critical one was a few modifications to the puppetlabs-mcollective module, which is why I’m currently using a git fork. PR’s have been raised for most of the changes, but they aren’t live yet.

• Extra Yaml in mco Needs more work.
• Remove Erlang dependency

So, this is the new blog. I’m hoping to talk about some of the things currently going on professionally, between puppet work, some inputs from #monitoringsucks responses, and various other things.

I’m probably going to flit from topic to topic, and may post some of my code that I’m working.

I’m also intersted in how to effectivly build new systes in a resilient fashion, inside AWS.

Somewhere in there, we may even have to build a dashboard for self service.

Coming up, in no particular order:

• heira_params – because I don’t want to bother with an ENC.
• puppet resillancy with a multi-AZ & multi-region app.
• App redundancy for the same.. or how to survive an AWS zone/region outage
• metrics, monitoring, and more.

If you have a slave that has a corrupted relay log, it will show: Last_Error: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave. in it’s mysqld.log & slave status.

This can happen because of a crash on the mysql slave, or a disk filling up, or something.

You can fix it by executing on the slave in question:

1. STOP SLAVE
2. SHOW SLAVE STATUS
3. Note down ‘Relay_Master_Log_File’ and ‘Exec_Master_Log_Pos’ entries.
4. RESET SLAVE
5. CHANGE MASTER TO ….. (use MASTER_LOG_FILE=relay_master_log_file and MASTER_LOG_POS=exec_master_log_pos from Step 3)
6. START SLAVE

For example, #5 might be something like:

change master to master_log_file="mysql-bin.001878",master_log_pos=898514732;

Not that this isn’t available elsewhere, but… just keeping it nicely stored somewhere for myself.

on ipa server:

ipa host-add host.example.com --password=MEEEEEP --force

on client:

ipa-client-install --hostname=host.example.com --domain=example.com --password=MEEEEEP --realm=EXAMPLE.COM --server=ipa.example.com --unattended --mkhomedir

1
2
3
4
5
6
7
8

#!/bin/bash
echo
echo "ipa host-add $1.example.com --password=MEEEEEP --force"
echo
echo
echo "ipa-client-install --hostname=$1.example.com --domain=example.com  --password=MEEEEEP --realm=EXAMPLE.COM --server=ipa.example.com --unattended --mkhomedir"

echo

Uninstall IPA

1
2
3
4
5

[root@solr05 ~]# ipa-client-install --uninstall -U
Unenrolling client from IPA server
Removing Kerberos service principals from /etc/krb5.keytab
Disabling client Kerberos and LDAP configurations
Restoring client configuration files

Run this on a ipa server. It’ll work there..

1. Find users with passwords that expire before DATE below

    ldapsearch -Y GSSAPI -h localhost -b "cn=users,cn=accounts,dc=ayisnap,dc=com" '(krbPasswordExpiration<=20140101000000Z)'>lsearch2`
   

2. Change the actual expiration date for those users to date :1/1/2015.

1
2
3
4
5
6
7
8
9
10

grep dn lsearch2  > dn
IFS=$'\n'
for i in  `cat dn`;
  do echo $i;
  echo "changetype: modify";
  echo "replace: krbpasswordexpiration";
  echo "krbpasswordexpiration: 20150101000000Z";
  echo ;
done > test
ldapmodify -D "cn=Directory Manager" -W -h localhost -f test

(If you have a version of FPM installed, you may want to remove & install the newest version before building…)

Building FPM:

1
2
3
4

gem install --install-dir /tmp/gems fpm
cd /tmp/gems/cache/

fpm -s gem -t rpm -d rpm-build -d rubygems -d ruby -d ruby-devel -d kernel-devel -d gcc -d gcc-c++ -d make -e fpm*.gem

Building other gems:

1

fpm -s gem -t rpm -d rubygems -d ruby *.gem

To install blockdiag:

yum install python-reportlab python-imaging
easy_install "blockdiag[pdf]"

Generate a pdf

nwdiag -Tpdf routing.diag -f /usr/share/fonts/dejavu/DejaVuSans.ttf

Generate a SVG

nwdiag -Tsvg routing.diag

Generate a png.  (Not as pretty, actually.)

nwdiag routing.diag

So, there’s a new way to use multiple versions of languages, such as Ruby, on RHEL, without breaking the system.  I’ve got this down, so I thought I should share it.

In this case, it’s replacing the need for RVM, and moving to a much more standardized system.  I don’t love can’t stand having GCC installed on production servers, so this is much better.

SCL – Software Collections :)

The goal of this was to get a recent version of Fog working on RHEL 6 / Centos 6.

First:  get the SCL repo:

1
2

cd /etc/yum.repos.d`
wget http://people.redhat.com/bkabrda/scl_ruby193.repo

That gives you the basics, but we also need to get some pieces for Fog, compiled for SCL. Thanks to the Katello Project, which is packaging Fog for use w/ Foreman, we can add their repo. You’ll have to make a repo for this, though:

http://fedorapeople.org/groups/katello/releases/yum/foreman-nightly/RHEL/6Server/x86_64/

—You could probably use that as your repo for ruby_193, even without the above scl repo.

This is what the repo file would look like:

1
2
3
4
5

[foreman-nightly]
name=Foreman Nightly
baseurl=http://fedorapeople.org/groups/katello/releases/yum/foreman-nightly/RHEL/6Server/x86_64/
enabled=1
gpgcheck=0

Now, install ruby_193:

yum install ruby193-ruby ruby193-rubygem-fog   ruby193-rubygem-rbvmomi

Now, you can use it:

scl enable ruby193 bash
which ruby

One thing, if you want to use ruby_193 inside scripts, you can use a stub for it:

/usr/bin/ruby193-ruby

Recently saw this at work:

Server with IPA (freeipa.org), running puppet 3.X (3.2.1, but not really relevant.)  Centos 6.4.

Normally

Puppet’s sshkey type  creates /etc/ssh/ssh_known_hosts by default.

SSH defaults to checking /etc/ssh/ssh_known_hosts & /etc/ssh/ssh_known_hosts2, if GlobalKnownHostsFile isn’t set inside ssh_config.   RHEL / Centos doesn’t have it in the default config file, so it’s probably using the default for you.

All very good.

BTW- they could fix a long time sshkey bug: https://projects.puppetlabs.com/issues/4145 – it defaults to mode 600, which really needs to be 644 to be useful.

With IPA

If you’re using IPA, though, with SSSD, it adds this to your /etc/ssh/ssh_config:

GlobalKnownHostsFile /var/lib/sss/pubconf/known_hosts

This in turn disables the defaults, and it only checks /etc/ssh/ssh_known_hosts2. Not /etc/ssh/ssh_known_hosts.

Fix is to use a target parameter, for the sshkey, to make sure you write to /etc/ssh/ssh_known_hosts2.

Not sure how I’m going to get a bug opened on this. It’s not a puppet bug at all. It’s probably a bug for OpenSSH, but I don’t have a binary that they build, so I can’t submit a bug :(

BTW: for those that are interested: strace ssh HOSTNAME was how I found the details.  Since it stops and waits for the human to agree to add the host key to known_hosts, it’s actually fairly easy to strace.