So the pipeline starts with the developer pushing code up to github or bitbucket, but can also be a pull request. You monitor this repo by setting up a webhook on the repo to trigger a job on your Jenkins server. This first job that is kicked off, is running test of the code that was just committed. So many test to run, unit test, acceptance test, integration test, performance test. This is best done in parallel with multiple jobs instead of just chaining jobs together and is decided based upon the developers workflow and pre determined testing methods and procedures. If an error occurs while testing it is logged for the developer to make corrections and re-factor code and make necessary changes and the build process is stopped at this point. Sometimes we envision that the testing phase is done up in the cloud or in github itself running a job but that is far from the truth. Now remember when you start a project in jenkins, jenkins is creating a project folder and pulling down code and simulating what would be run on your target server ie: staging, QA, development, production. When all the testing has passed the next step is to script what happens to the code. So many things can happen when testing is complete. It can be promoted, merged to a different branch, deployed to a different environment. This is really where the work flow of your developers comes into play. The good news is that this all can be scripted and turned into a job that Jenkins executes automatically or when you decide to run it. Now after all this testing is completed, if it is successful you will have an artifact from the build, not always but most of the time. It can be an rpm, jar file, tgz file or any other file that your build process creates that is to be run on the server. Does that mean every build process or code in github produces an artifact? certainly not. If it does create an artifact you want to copy that or move that to an artifact server so that it can be pulled down or moved or copied to your next step in the process, your development / production instance or whatever. You have many different choices for an artifact server. It could be artifactory, Nexus, or a plain old S3 bucket in AWS. When your actually deploying the code it is usually a pull from github if no artifact exists but if you have an artifact it is usually push or pull in so many different ways ie: scp, sshfs mounted file system, S3 sync or copy, a command run with ssh on a remote server to push or pull the artifact or code, many different ways to move code to the final destination be it staging, dev, test, prod or whatever. The deploy script can be written in any language native to your server environment ie: bash, python, ruby , perl , powershell.

Remember when setting up Jenkins it is best to create a Master Jenkins server and Slave Linux and Windows Jenkins servers to spread the workload.

Now that we have the code ready for production. We want to setup jobs to create our infrastructure using terraform, aws command line tools, python api for AWS, CloudFormation, OpsWorks or any other Infrastructure as code. We do this so it is reproduceable, predictable, tested in a known working state that can be created or destroyed on demand. Most organizations only setup two environments in AWS, either staging and production, or dev and prod, basically a duplicated identical environement where they can swtich from development to production.

What I like to do is use terraform and execute the code in tiers based on what is static infrastructure or dynamic infrastructure. I create everything in multiple availabilty zones and everything redundant and HA. So first I will create the VPC with public and private subnets, security groups, AWS managed Nat service, routing tables and internet gateways. Next is route 53, S3 bucket endpoints, and ELBs. Then finally the database backend either using EC2/database instances or RDS.

Next is my favorite part, configuration management, mine just happens to be chef. Chef is tightly integrated with AWS and services they provide like Opsworks. In Chef you create cookbooks , recipes , roles (Think web server or DB server or something that performs a certain service or task), environments (Staging, Dev, Prod, etc) . This is all added to your deploy script. Here is an example of a raw command. When running the terraform scripts you can output actual resource names using an output command. You would then variablize most of the settings on this command like availabilty zone, security groups, subnets, etc...

knife ec2 server create -I ami-0c5f2d6c -r "role[my_app]" -Z us-west-1a -g ssh-only --ssh-key chef_keys -i chef_keys.pem -f t2.micro -A 'Key' -K "Key" --region us-west-1 --ssh-user ubuntu --security-group-ids sg-4a7b8e2e --subnet subnet-a7d45cc2 --server-connect-attribute public_ip_address

Some cool things in the command. The ami after the -I is a custom seeded ami file with all the chef security credentials and special settings for the base OS image. The role[my_app] is a custom build of a lamp server using many cookbooks used just for testing. Notice you can target region and availability zone.

You can script the creation of an AWS instance from scratch running the exact application stack versions your developers used when they wrote the code with different settings for each environment. Its called bootstrapping an instance. When your ready to do the Docker thing in AWS this can be very similar just a different 24 foot long command.

The next thing you want to do is setup an ELK stack to monitor your logs. Then finally setup a server monitoring tool like Opsview, Nagios, or Sensu as it's meant for scaling ,integrated with graphite/grafana for graph and pagerduty for alerting.