Jan 6, 2015 - Legal Television and Movies for Expats, Part 2

A while ago,I wrote a blog post about legal television and movies for expats, using streaming services combined with a VPN.

Unfortunately it looks like things are changing: movie studios are fighting back against our outrageous attempt to give them our money, by blocking these VPN services.

No doubt, if this escalates, they’ll certainly find the range my current VPN provider uses and block it, and I’ll be stuck with what’s available in Germany. Thankfully, the situation has improved a lot - Netflix and Amazon both stream in Germany now, but still mostly crappy dubbed content. A very limited selection of Original Version (OV) content is available that falls far short of what’s streamable in the U.S.

Most television shows take years to become available here due to complex localization and licensing requirements. Movies take less time, but still not on par with the U.S., Sony completely ignored the rest of the world when they launched The Interview, for example.

The internet has no borders. Please stop building artificial ones. :-(

Nov 11, 2013 - Foreman FreeIPA Integration Guide

Note! In Foreman 1.5, FreeIPA realm join integration is now built-in!

Two projects that I'm really loving at the moment are The Foreman and FreeIPA. The Foreman is lifecycle management tool for physical and virtual servers (think Cobbler on PCP), and FreeIPA provides central authentication: directory services, kerberos, policy enforcement, and a PKI infrastructure. Why not glue them together? This is my first attempt, and it's all a bit manual and unpolished. There's an effort to get this integration into the Foreman Smart Proxy itself.

All of the scripts here are in a Github Gist.

The goals here are:
  • Foreman authenticates against FreeIPA
  • Signed Certificates for Foreman, Puppetmaster, and Clients
  • New Hosts automatically register to IPA and get a signed certificate
  • A host gets deleted from IPA when it is deleted from Foreman


  • Installed Foreman Server on a Red Hat-based distro (RHEL, CentOS, Fedora, et al).
  • Foreman server is registered to FreeIPA
  • Installed FreeIPA (or Red Hat IdM) Server


  1. Foreman LDAP Authentication
  2. FreeIPA PKI Infrastructure
  3. Automatic IPA Registration/Deletion

Foreman LDAP Authentication

End Goal: Users can login to the Foreman using FreeIPA credentials. Individual access rights still need to be granted in the Foreman GUI itself, though. Based on the FreeIPA Ejabberd Integration Guide
  1. Create a foreman.ldif file, replacing dc=bitbin,dc=de with your DN, and providing an appropriately secure password:
    dn: uid=foreman,cn=sysaccounts,cn=etc,dc=bitbin,dc=de
    changetype: add
    objectclass: account
    objectclass: simplesecurityobject
    uid: foreman
    userPassword: 8j926SEpcOvM0WLI
    passwordExpirationTime: 20380119031407Z
    nsIdleTimeout: 0
  2. Import the LDIF (change localhost to an IPA server if needed), you'll be prompted for your Directory Manager password:
    # ldapmodify -h localhost -p 389 -x -D \
    "cn=Directory Manager" -W -f foreman.ldif
  3. Add an IPA group for foreman_users (optional):
    # ipa group-add --desc="Foreman Users" foreman_users
  4. Now login to the Foreman as an Admin, click on "LDAP Authentication" under More/Users. Then click New LDAP Source and fill in the details, changing dn's where appropriate to your own domain:

    • Server: astriaporta.bitbin.de
    • Port: 636
    • TLS: checked
    • Account username: uid=foreman,cn=sysaccounts,cn=etc,dc=bitbin,dc=de
    • Account password: 8j926SEpcOvM0WLI
    • Base DN: cn=accounts,dc=bitbin,dc=de
    • Filter (optional): (memberOf=cn=foreman_users,cn=groups,cn=accounts,dc=bitbin,dc=de)
    • Automatically create accounts in the Foreman: checked
    • LDAP mappings are as the examples given.
    </li> </ol>

    FreeIPA Certs for Foreman + Puppet

    End Goal: Foreman and the Puppetmaster use certificates from the FreeIPA server.
    1. Backup the existing SSL directory:
      # mv /var/lib/puppet/ssl /var/lib/puppet/ssl.old
    2. Make the appropriate directory structure:
      # mkdir -p /var/lib/puppet/ssl/{private_keys,certs}
    3. For the next steps you need to have a Kerberos ticket for a user with sufficient privileges (e.g. admin)
      # kinit admin
    4. Create the service principal:
      # ipa service-add puppet/`hostname`
    5. Request certificates from the IPA server
      # ipa-getcert request -K puppet/`hostname` -D `hostname` \
       -k /var/lib/puppet/ssl/private_keys/`hostname`.pem \
       -f /var/lib/puppet/ssl/certs/`hostname`.pem
    6. Check on the request, you should see the status as MONITORING if successful:
      # ipa-getcert list
      Request ID '20131106211000':
          status: MONITORING
          stuck: no
          key pair storage: type=FILE,location='/var/lib/puppet/ssl/private/gatebuilder.bitbin.de.pem'
          certificate: type=FILE,location='/var/lib/puppet/ssl/certs/gatebuilder.bitbin.de.pem'
          CA: IPA
          issuer: CN=Certificate Authority,O=BITBIN.DE
          subject: CN=gatebuilder.bitbin.de,O=BITBIN.DE
          expires: 2015-11-07 21:10:01 UTC
          eku: id-kp-serverAuth,id-kp-clientAuth
          pre-save command:
          post-save command:
          track: yes
          auto-renew: yes
    7. Copy the IPA CA.cert:
      # cp /etc/ipa/ca.crt /var/lib/puppet/ssl/certs/ca.pem
    8. Take a peek in the SSL directories, and you'll see our new certs:
      # ls  {private_keys,certs}
      ca.pem  gatebuilder.bitbin.de.pem
    9. Make sure permissions are sensible:
      # chown -R puppet:puppet /var/lib/puppet/ssl
      # chmod 600 /var/lib/puppet/ssl/{private_keys,certs}/`hostname`.pem
    10. Edit /etc/puppet/puppet.conf:
      • Add to [main]:
        # This disables the CRL.  I need to fix this at a
        # later time
        certificate_revocation = false
      • In [master], change:
        ca = false
      • Restart httpd (Foreman-configured Puppet runs in Passenger):
        # service httpd restart
    11. Browse to the Foreman, and you should see it using the new SSL certificates signed by your IPA CA. Ideally you should import the IPA CA.crt on your local box and trust it.
    12. </ol>

      Registration at Provision-Time

      The idea here is that our machines when foreman creates them are automatically registered to FreeIPA with a one-time password, and if later deleted in the Foreman, they are removed from FreeIPA too. Hosts also get an SSL certificate signed by the FreeIPA server to talk to puppet. The flow looks like this: Here's a really confusing graphic that may or may not make things clearer

      Creating IPA User with Right Permissions

      A previous version of this guide called this user "foreman" - don't do that, it'll interfere with upgrading later, as the RPM packaging expects to use a local user named "foreman."
      1. Create the user:
        # kinit admin
        Password for admin@BITBIN.DE:
        # ipa user-add --first="The" --last="Foreman" foreman_reg \
        Enter Password again to verify:
        Added user "foreman_reg"
      2. Grant host enrollment privileges:
        # ipa role-add-member --users=foreman_reg "Host Enrollment"
      3. We need to modify the Host Enrollment role to actually allow the Foreman user to add brand new hosts and delete them too -- so Foreman can completely manage the machine lifecycle.
        # ipa privilege-add-permission 'Host Enrollment' \
        --permissions='Add Hosts'
        # ipa privilege-add-permission 'Host Enrollment' \
        --permissions='Remove Hosts'
      4. Change foreman password after first time:
        [root@gatebuilder ~]# kinit foreman_reg
        Password for foreman_reg@BITBIN.DE:
        Password expired.  You must change it now.
        Enter new password:
        Enter it again:

        Configuring the Create/Destroy Hook

        Grab the scripts from the Github Gist.
        1. On Foreman, install the hooks gem:
          # yum -y install ruby193-rubygem-foreman_hooks
        2. And make the directory structure we need:
          # mkdir -p /usr/share/foreman/config/hooks\
        3. Put foreman-ipa into /etc/sysconfig/ with the right permissions
          # chown foreman /etc/sysconfig/foreman-ipa
          # chmod 600 /etc/sysconfig/foreman-ipa
        4. Configure /etc/sysconfig/foreman-ipa
          # Are we using IPA as the CA?
          # Allow Foreman to delete hosts from IPA
          # Hostname of an IPA server
          # User with appropriate permissions
          # Foreman API User/Password
        5. Put 10_integrate_freeipa.sh into /usr/share/foreman/config/hooks/host/managed/create and create a symlink to destroy and after_commit:
          # ln -s /usr/share/foreman/config/hooks\
          /host/managed/create/10_integrate_freeipa.sh ../destroy
          # ln -s /usr/share/foreman/config/hooks\
          /host/managed/create/10_integrate_freeipa.sh ../after_commit
        6. Restart foreman to get it to notice the new hooks:
          # service foreman restart
          # service httpd restart
        7. Take a look at the logs to make sure the hooks were registered, look in /var/log/foreman/production.log:
          Finished registering 1 hooks for Host::Managed#destroy
          Finished registering 1 hooks for Host::Managed#after_commit
          Finished registering 1 hooks for Host::Managed#create
        8. The last step in this is integrating into your provisioning template. You'll need to get ipa-client installed in your packages list, and remove the other puppet registration thingy from the Foreman. I have a snippet that looks like this:
          # Register to IPA, two times
          # in case of https://fedorahosted.org/freeipa/ticket/3377
          ipa-client-install --mkhomedir -w <%= @host.params['ipa_onetime'] %> -f -U
          ipa-client-install --mkhomedir -w <%= @host.params['ipa_onetime'] %> -f -U
          # Make Puppet Certificate Directories
          mkdir -p /var/lib/puppet/ssl/{private_keys,certs}
          # Generate IPA Certificate
          ipa-getcert request -K puppet/<%= @host.name %>  -D <%= @host.name %> \
          -k /var/lib/puppet/ssl/private_keys/<%= @host.name %>.pem \
          -f /var/lib/puppet/ssl/certs/<%= @host.name %>.pem
          # Workaround for "stack too deep" problem
          # http://projects.puppetlabs.com/issues/21869
          cp /etc/ipa/ca.crt /var/lib/puppet/ssl/certs/ca.pem
          cat <<EOF > /etc/puppet/puppet.conf
              # The Puppet log directory.
              # The default value is '$vardir/log'.
              logdir = /var/log/puppet
              # Where Puppet PID files are kept.
              # The default value is '$vardir/run'.
              rundir = /var/run/puppet
              ssldir = /var/lib/puppet/ssl
              server = <%= @host.puppetmaster %>
              # The file in which puppetd stores a list of the classes
              # associated with the retrieved configuratiion.  Can be loaded in
              # the separate ``puppet`` executable using the ``--loadclasses``
              # option.
              # The default value is '$confdir/classes.txt'.
              classfile = $vardir/classes.txt
              # Where puppetd caches the local configuration.  An
              # extension indicating the cache format is added automatically.
              # The default value is '$confdir/localconfig'.
              localconfig = $vardir/localconfig
              certificate_revocation = false
              certname = <%= @host.name %>
          puppet agent --test
          chkconfig puppet on
        9. When the machine boots it will use one-time password authentication with FreeIPA and grab an SSL certificate for use with Puppet. And you get the bonus of when you delete the machine in Foreman, it gets deleted in IPA too.
        10. </ol>

Nov 6, 2013 - A Gerrit Workflow for Puppet Environments

Managing Puppet manifests in source control is nothing new. Puppet Labs has their own suggested workflow that uses a git post-receive hook to automatically publish a git branch as an environment. It’s a clever idea, but not one that I think scales well with a bigger team. It would be better if manifests could go through code review, and be approved before going live.

This is the perfect use for Gerrit: any proposed changes go into a pending queue, and are only merged after a review process.

In the Puppet workflow with Gerrit, committers push their proposed changes to the pending change area for master, and then they go through review. Once approved, they are committed to master. When ready to move on to another environment, they can be promoted – by way of gerrit – by merging to development, then quality, then production for example. Life cycle management for your puppet modules.


The diagram below shows how it works: a developer gets an authoritative copy of the repo, and makes some changes, but instead of committing back to the authoritative branch, it is committed to a special staging area. The staged commit shows up in the Gerrit code review GUI. Members of the team review the code, and provide any relevant feedback. Once the code gets a +2 vote, it’s merged into the authoritative repo.

Figure 1

When a change is merged, the change-merged hook is executed. I’ve written one that will automatically publish the puppet environment. Unfortunately we need this separate hook, because gerrit does not look at anything in the .git/hooks directory.

Now, an interesting thing about gerrit is that all of the reviewers do not need to the human. Using the Gerrit Trigger for Jenkins, a staged commit can be tested, run through puppet-lint, or any other steps that are needed. The Jenkins wiki explains in more detail how it works. Basically, Jenkins can submit a vote back based on it’s results. Assuming the tests were successful, reviewer(s) can examine the proposed changes – and once the final approval is given, it’s deployed instantly to your puppetmaster.