How to set-up a simple web development environment (web & database server) with Puppet

ecs-admin 23rd August 2016

In this step-by-step guide, we will see how to set-up a Puppet Master in Amazon Web Services, and how to use it to create two other AWS instances.

We will then use Puppet to configure these two instances, one will be a MySQL Database Server and the other an Apache Web Server.

Summary

  1. AWS Setup
  2. Set up the Master
    1. Create the AWS Master Instance
    2. Install Puppet Enterprise
  3. Configure the Agent Nodes
    1. Launch the agent nodes with Puppet
    2. Configure Apache and MySQL using Roles and Profiles
      1. Create The Database and and the Webserver Roles
      2. Create the Apache and MySQL Profiles
    3. Classify our Nodes
    4. Manually run Puppet on each Agent Nods
  4. Related Articles

AWS Setup

Note that in this Guide, we use the eu-central-1 / Frankfurt zone.

If you intend to use a different zone, you will have to change the ami-id in the appropriate places in the scripts.

  1. Login to your AWS EC2 Console.
  2. Select the zone mentioned above.
  3. Create a AWS Access Key and a Secret Key in Security Credentials > Users > Your User Name > Create Access Key, and keep them handy so you can refer to them later.
  4. Go to the EC2 console.
  5. Create a Key Pair in Network & Security > Key Pairs > Create Key Pairs called Webdev-forest,and save the pem file to an accessible locaiton. (You will need this in order to access the Master).

Set up the Master

Create the AWS Master Instance

  1. In order to create the Master Instance, select EC2 Console > Instances > Launch Instance, and configure it as follows:
    1. Choose the Ubuntu Server 14.04 LTS (HVM), SSD Volume Type AMI.
    2. Choose the t2.large type.
    3. Use the default instance details settings.
    4. Use the default storage, 20 GB SSD.
    5. Give it a recognizable name, e.g. master_of_puppets.
    6. Create a security group with ports 22 for SSH only from your IP, and 3000, 8140, 443, 61613, 8142 for puppet services from anywhere.
    7. Review and launch.
    8. Use the keypair that you just created.
    9. Launch!
    10. Wait until the instance has finished initializing.

Install Puppet Enterprise

  1. Now using the key created before and the public hostname of your instance, which you can find in the ec2 description of your instance at the Public DNS section
    1. chmod 400 Webdev-forest.pem
    2. ssh -i Webdev-forest.pem ubuntu@[public hostname]
    3. accept the connection 
  2. Become root
    1. sudo su 
  3. Edit your /etc/hosts and add the following line at the top:
    1. vim /etc/hosts
    2. "127.0.0.1   localhost master.puppet.vm master puppet" 
  4. Change the hostname to “master.puppet.vm”
    1. hostname master.puppet.vm 

  5. Download the pe master installer 
    1. wget -O puppet-installer.tar.gz "https://pm.puppetlabs.com/cgi-bin/download.cgi?dist=ubuntu&rel=14.04&arch=amd64&ver=latest"
  6. Unpack the installer
    1. tar -xf puppet-installer.tar.gz
  7. Install puppet master
    1.  ./puppet-enterprise-<version>-ubuntu-14.04-amd64/puppet-enterprise-installer
    2. Select the [1] option to perform a guided installation
    3. Copy the public hostname of your ec2 instance, and go to https://<public-hostname>:3000Puppet_1.png
    4. There will be an error displayed by your browser, add an exception in firefox or click on advanced and then proceed in chrome. For a more elaborate guide go tohttps://docs.puppetlabs.com/pe/latest/console_accessing.html to access the console.
    5. Click on Let’s get started!
    6. Select a monolithic installation
    7. Type in the Puppet master FQDN: master.puppet.vm
    8. Type in the Puppet master DNS aliases: puppet
    9. Type in a Console Administrator password. Later on you will use it to login as the adminuser.
    10. Click on Submit and then Continue
    11. Now the Puppet Installer will do some checks before the installation, and will probably prompt some warnings which can be skipped.
    12. Click Deploy Now
    13. This step will take around 10 minutes, which is normal and you will then see this screen indicating that all went well:Puppet_2.png
  8. access the console at https://<public-hostname>
    1. The user is “admin” and the password is the one that you chose in the step before.
    2. You will then see the console:Puppet_3.png

The puppet master is now all set, so let’s take care of the agents.

Configure the Agent Nodes

Launch the agent nodes with Puppet

  1. On the master, create a new directory called create_instances in the /tmp/ directory.
    1. mkdir ~/create_instances 
  2. Create a new file create.pp that will create the instances
    1. vim ~/create_instances/create.pp
    2. Paste the following code: 
      $pe_master_hostna$::fqdn # Get the master's fqdn
me = $facts['ec2_metadata']['hostname'] # Get the hostname of the master $pe_master_ip = $facts['ec2_metadata']['local-ipv4'] # Get the ip of the master $pe_master_fqdn = 
# Set the default for the security groups
Ec2_securitygroup {
  region => 'eu-central-1', # Replace by the region in which your puppet master is
  ensure => present,
  vpc    => 'My VPC', # Replace by the name of your VPC
}

# Set the default for the instances
Ec2_instance {
  region        => 'eu-central-1', # Replace by the region in which your puppet master is
  key_name      => 'Webdev-forest', # Replace by the name of your key if you chose something else
  ensure        => 'running',
  image_id      => 'ami-87564feb', # ubuntu-trusty-14.04-amd64-server-20160114.5 (ami-87564feb)
  instance_type => 't2.micro',
  tags          => {
    'OS'    => 'Ubuntu Server 14.04 LTS',
    'Owner' => 'Michel Lebeau' # Replace by your name
  },
  subnet        => 'My Subnet', # Replace by the name of your Subnet
}

# Set up the security group for the webserver
ec2_securitygroup { 'web-sg':
  description => 'Security group for web servers',
  ingress     => [{ 
    # Open the port 22 to be able to SSH into, replace by your.ip/32 to secure it better 
    protocol => 'tcp',
    port     => 22,
    cidr     => '0.0.0.0/0'
  },{ 
    # Open the port 80 for HTTP
    protocol => 'tcp',
    port     => 80,
    cidr     => '0.0.0.0/0'
    },
  ],
}

# Set up the security group for the database server
ec2_securitygroup { 'db-sg':
  description => 'Security group for database servers',
  ingress     => [{ 
    # Open the port 22 to be able to SSH into, replace by your.ip/32 to secure it better 
    protocol => 'tcp',
    port     => 22,
    cidr     => '0.0.0.0/0'
  },{ 
    # Open the port 3306 to be able to access mysql
    protocol => 'tcp',
    port     => 3306,
    cidr     => '0.0.0.0/0'
    },
  ],
}
 
# Set up the instances, assign the security groups and provide user data that will be executed at the end of the initialization 
ec2_instance { 'webserver':
  security_groups => ['web-sg'],
  user_data       => template('/root/create_instances/templates/webserver.sh.erb'),
}
ec2_instance { 'dbserver':
  security_groups => ['db-sg'],
  user_data       => template('/root/create_instances/templates/dbserver.sh.erb'),
}

      You can find the VPC and subnet in the VPC section of AWS, please note that Puppet expects the name of the VPc and subnet, the ID will not work.

    3. If you are using a different region than eu-central-1, change the region and the image_id accordingly.
  3. Create 2 templates
    1. Create a directory called “templates” inside the /tmp/create_instances directory
      1. mkdir ~/create_instances/templates
    2. Create the webserver template
      1. vim ~/create_instances/templates/webserver.sh.erb 
        #!/bin/bash
PE_MASTER='<%= @pe_master_hostname %>'
echo "<%= @pe_master_ip %> <%= @pe_master_fqdn %>" >> /etc/hosts
# Download the installation script from the master and execute it
curl -sk https://$PE_MASTER:8140/packages/current/install.bash | /bin/bash -s agent:certname=webserver
    3. Create the dbserver template
      1. vim ~/create_instances/templates/dbserver.sh.erb  
        #!/bin/bash
PE_MASTER='<%= @pe_master_hostname %>'
echo "<%= @pe_master_ip %> <%= @pe_master_fqdn %>" >> /etc/hosts
# Download the installation script from the master and execute it
curl -sk https://$PE_MASTER:8140/packages/current/install.bash | /bin/bash -s agent:certname=dbserver
    4. Now let’s create the instances:
      1. Install the retries gem and the Amazon AWS Ruby SDK gem
        1. /opt/puppetlabs/puppet/bin/gem install aws-sdk-core retries
      2. export your aws access key, here is a very small guide on where to find it: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html
        1. mkdir ~/.aws/
        2. vim ~/.aws/credentials 
          [default]
aws_access_key_id =          # Paste here your Access Key ID
aws_secret_access_key =   # Paste here your Secret Access Key ID
region =                                # Specify your region, optional
      3. install puppet’s AWS module
        1. puppet module install puppetlabs-aws
      4. finally apply the create script
        1. puppet apply /root/create_instances/create.pp 
          [root@master ~]# puppet apply /root/create_instances/create.pp
Notice: Compiled catalog for master.puppet.vm in environment production in 0.11 seconds
Notice: /Stage[main]/Main/Ec2_instance[webserver]/ensure: changed absent to running
Notice: /Stage[main]/Main/Ec2_instance[dbserver]/ensure: changed absent to running
Notice: Applied catalog in 25.15 seconds
      5. Wait for the instances to be started and initialized. Once this process is finished, puppet will run and you will have to accept their certificates before they can communicate with the master.Puppet_4.png
      6. In the Puppet Enterprise Console, go to Nodes > Unsigned certificatesPuppet_5.png

         

      7. Accept all so the nodes will be able to get their latest configuration from the master.

Configure Apache and MySQL using Roles and Profiles

Now, we have two running Puppet Agent nodes communicating with our Puppet Enterprise Master. Only a few steps more and we will enjoy our new website!

Create The Database and and the Webserver Roles

The Roles will define the business logic of our applications, and will be composed by one or more profiles.

  1. In the master, navigate to the production environment:
    1. cd /etc/puppetlabs/code/environments/production/ 
  2. create the modules/roles/manifests directory
    1. mkdir -p modules/roles/manifests 
  3. create the dbserver role
    1. vim modules/roles/manifests/dbserver.pp 
      # Role for a Database Server
class roles::dbserver {
  # Include the mysql profile
  include profiles::mysql
}
  4. create the webserver role
    1. vim modules/roles/manifests/webserver.pp 
      # Role for a Web Server
class roles::webserver {
  # Include the apache profile
  include profiles::apache
}

Create the Apache and MySQL Profiles

 

Now, we will create our Profiles which will define the application stack for Aache and MySQL

  1. create the modules/profiles/manifests directory
    1. mkdir -p modules/profiles/manifests 
  2. create the apache profile
    1. vim modules/profiles/manifests/apache.pp 
      # Install and configure an Apache server
class profiles::apache {
  # Install Apache and configure it
  class { 'apache':
    mpm_module => 'prefork',
    docroot    => '/var/www',
  }
  # Install the PHP mod
  include apache::mod::php

  # Install php5-mysql for PDO mysql in PHP
  package { 'php5-mysql':
    ensure => installed,
  }

  # Get the index.php file from the master and place it in the document root
  file { '/var/www/index.php':
    ensure => file,
    source => 'puppet:///modules/profiles/index.php',
    owner  => 'root',
    group  => 'root',
    mode   => '0755',
  }

  # Declare the exported resource
  @@host { 'webserver':
    ip           => $::ipaddress,
    host_aliases => [$::hostname, $::fqdn] ,pin
  }

  # Collect the exported resources
  Host <<||>>
}
  3. create the mysql profile
    1. vim modules/profiles/manifests/mysql.pp 
      # Install and configure a MySQL server
class profiles::mysql {

  # Install MySQL Server and configure it
  class {'mysql::server':
    root_password           => 'p4ssw0rd',
    remove_default_accounts => true,
    restart                 => true,
    override_options        => {
      mysqld => {
        bind_address            => '0.0.0.0',
        'lower_case_table_name' => 1,
      }
    }
  }
  # Copy the sql script from the puppet master to the /tmp directory
  file { 'mysql_populate':
    ensure => file,
    path   => '/tmp/populate.sql',
    source => 'puppet:///modules/profiles/populate.sql',
  } ->
  # Only once the file has been copied, use it to populate a new database
  mysql::db { 'cats':
    user     => 'forest',
    password => 'p4ssw0rd2',
    grant    => ['SELECT', 'UPDATE', 'INSERT', 'DELETE'],
    host     => '%', # You can replace by 'webserver' to make it more secure,
    # but you might have to flush your hosts in mysql for it
    # to be taken into account
    sql      => '/tmp/populate.sql',
  }

  # Declare the exported resources
  @@host { $::hostname:
    ip           => $::ipaddress,
    host_aliases => [$::fqdn, 'database'] ,
  }

  # Collect the exported resources
  Host <<||>>
}
  4. Create the files that will be used to pre populate the MySQL database with some sample data and the webpage that will consume that information
    1. mkdir modules/profiles/files
    2. vim modules/profiles/files/populate.sql

      1. USE `cats`;

CREATE TABLE `family` (
  `id` mediumint(8) unsigned NOT NULL auto_increment,
  `Name` varchar(255) default NULL,
  `Age` mediumint default NULL,
  PRIMARY KEY (`id`)
) AUTO_INCREMENT=1;

INSERT INTO `family` (`Name`,`Age`) VALUES ("Hasad",6),("Uma",5),("Breanna",17),("Macaulay",14),("Colton",11),("Serina",16),("Emery",13),("Christian",7),("Vladimir",16),("Wang",13);
INSERT INTO `family` (`Name`,`Age`) VALUES ("Hermione",12),("Yoshio",9),("Hilel",10),("Autumn",6),("Solomon",7),("Briar",6),("Armand",9),("Alyssa",1),("Shelby",1),("Yasir",15);
INSERT INTO `family` (`Name`,`Age`) VALUES ("Wallace",1),("Yoshio",5),("Pascale",6),("Dalton",17),("Trevor",9),("Joan",10),("Zephr",14),("Neville",3),("Nicole",4),("Halee",14);
INSERT INTO `family` (`Name`,`Age`) VALUES ("Wayne",15),("Maile",8),("Alfonso",9),("Neve",6),("Heidi",16),("Mona",11),("Mollie",16),("Audra",16),("Karyn",12),("Acton",17);
INSERT INTO `family` (`Name`,`Age`) VALUES ("Xyla",1),("Cole",6),("Blossom",9),("Sybill",4),("Lavinia",4),("Keely",14),("Gwendolyn",15),("Trevor",10),("Acton",12),("Christine",10);
INSERT INTO `family` (`Name`,`Age`) VALUES ("Stone",17),("Erich",12),("Elijah",10),("Emerson",14),("Rafael",8),("Scott",17),("Olympia",13),("Nehru",14),("Casey",8),("Michael",3);
INSERT INTO `family` (`Name`,`Age`) VALUES ("Montana",8),("Heidi",11),("Edward",13),("Xenos",1),("Venus",9),("Malik",5),("Madeline",2),("Sacha",8),("Whitney",13),("Eagan",8);
INSERT INTO `family` (`Name`,`Age`) VALUES ("Lewis",2),("Guinevere",17),("Oliver",6),("Jana",7),("Rachel",2),("Ariel",7),("Pamela",6),("Medge",11),("Clare",10),("Meghan",8);
INSERT INTO `family` (`Name`,`Age`) VALUES ("Stone",10),("Chase",4),("Vladimir",17),("Grace",11),("Damon",15),("Ferdinand",11),("Veronica",14),("Wesley",13),("Zelda",15),("Eugenia",6);
INSERT INTO `family` (`Name`,`Age`) VALUES ("Carlos",9),("Cherokee",14),("Theodore",3),("Tanisha",11),("Grant",7),("Xyla",6),("Austin",11),("Madison",4),("Kasper",7),("Andrew",10);
    3. vim modules/profiles/files/index.php 
      <?php
echo "<h1>Our small cat family</h1>";
echo "<table style='border: solid 1px black;'>";
echo "<tr><th>Id</th><th>Name</th><th>Age</th></tr>";

class TableRows extends RecursiveIteratorIterator {
    function __construct($it) {
        parent::__construct($it, self::LEAVES_ONLY);
    }

    function current() {
        return "<td style='width:150px;border:1px solid black;'>" . parent::current(). "</td>";
    }

    function beginChildren() {
        echo "<tr>";
    }

    function endChildren() {
        echo "</tr>" . "\n";
    }
}

$host = "database";
$port = "3306";
$username = "forest";
$password = "p4ssw0rd2";
$dbname = "cats";

try {
    $conn = new PDO("mysql:host=$host;port=$port;dbname=$dbname", $username, $password);
    $conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
    $stmt = $conn->prepare("SELECT id, Name, Age FROM family");
    $stmt->execute();

    // set the resulting array to associative
    $result = $stmt->setFetchMode(PDO::FETCH_ASSOC);
    foreach(new TableRows(new RecursiveArrayIterator($stmt->fetchAll())) as $k=>$v) {
        echo $v;
    }
}
catch(PDOException $e) {
    echo "Error: " . $e->getMessage();
}
$conn = null;
echo "</table>";
?>
  5. install the apache and mysql modules
    1. puppet module install puppetlabs-apache
    2. puppet module install puppetlabs-mysql

 

Classify our Nodes

  1. Edit the manifest/site.pp
    1. vim manifests/site.pp 
      node 'dbserver'{
  include roles::dbserver
}

node 'webserver'{
  include roles::webserver
}

node default {
}

 

Manually run Puppet on each Agent Node

Puppet can be run using various methods, with the CLI, using MCollective or by using the Web Console for example. In this case we are going to use MCollective:

root@master:~# su - peadmin

peadmin@master:~$ mco puppet runonce -v -I webserver -I dbserver 

 * [ ============================================================> ] 2 / 2


webserver                               : OK
    {:summary=>      "Started a Puppet run using the '/opt/puppetlabs/bin/puppet agent --onetime --no-daemonize --color=false --show_diff --verbose --splay --splaylimit 120' command",     :initiated_at=>1471353250}

dbserver                                : OK
    {:summary=>      "Started a Puppet run using the '/opt/puppetlabs/bin/puppet agent --onetime --no-daemonize --color=false --show_diff --verbose --splay --splaylimit 120' command",     :initiated_at=>1471353250}



---- rpc stats ----
           Nodes: 2 / 2
     Pass / Fail: 2 / 0
      Start Time: 2016-08-16 13:14:11 +0000
  Discovery Time: 0.00ms
      Agent Time: 142.88ms
      Total Time: 142.88ms
peadmin@master:~$

To check if Puppet run successfully in the nodes and the changes that were applied to them, login to the Web Console and go to Configuration > Overview

 

Puppet_7.png

Now paste the public address of your webserver in your favourite browser and voilà, you are done! Note that if you get a Error: SQLSTATE[HY000] [2005] Unknown MySQL server host ‘database’ (2), you should try to run puppet using mco another time, as the exported resources haven’t been collected. This happens if the ip from the dbserver is not exported before when the webserver collects its resources. Running puppet another time will collect it.

Please note that if you terminate an AWS instance and start another with the create.pp script, it will have the same certname as the one that has been terminated, however the IP will differ. In order for Puppet to run correcty in this case, on the master execute:

puppet cert clean <certname>
With <certname> being either dbserver or webserver,

Related articles

  1. https://forge.puppetlabs.com/puppetlabs/aws
  2. https://forge.puppetlabs.com/puppetlabs/mysql
  3. https://forge.puppetlabs.com/puppetlabs/apache
  4. https://docs.puppetlabs.com/puppet/latest/reference/modules_fundamentals.html

Puppet Enterprise is one of the leading continuous delivery technologies, building on its heritage in infrastructure automation with the addition of Puppet Application Orchestration. Forest Technologies are proud partners of Puppet Labs and experts in delivering rapid value to our customers’ digital transformation initiatives using Puppet Enterprise.

Found this interesting? Why not share it: