Archive for the ‘Continuous integration’ Category

Dell EMC Dojo at Hopkinton! Reviewing what we covered, and what we learned

Reviewing what we covered, and what we learned

Hey again everyone! We’re writing out today to talk about a few topics that we covered during our time at the Dell EMC Dojo Days in Hopkinton. We met with a lot of great minds and took plenty of input into how we work. We also gave a lot of insight to passersby in what we did and how we worked.

Firstly, Brain Roche and Megan Murawski debuted the famous “Transformation Talk”. We normally give this presentation in preparation for an engagement with another team, but in this case, it was given to allow open criticism and circulation of our methodology to the rest of the company. We cover in this presentation: pairing, and why it’s important to our process, why we use TDD (Test Driven Development) (and why you should too!), and our weekly meetings including Retros, IPM, and Feedback to name a few. We had plenty of great ideas and questions, as usual, and we realized twenty minutes over time that we couldn’t get Brian off the stage.

Xuebin He eventually got Brian off-stage for a talk he conducted on CICD (Continuous Integration, and Continuous Deployment). Xuebin being one of the developers at the Dojo allowed him to be a bit more technical in his talk, and cover some of the programming practices and tools we use to achieve this at the Dojo. Concourse is our tool for running our beautifully constructed tests, along with standard mocking design patterns and the code quality produced with TDD.

We picked up again on Tuesday at 1145 to talk about why a PaaS exists, and why it’s important. That talk, given by yours truly, was focused on some of the common technical roadblocks that keep developers, customers, and managers from being able to work efficiently; as well as the ways using a PaaS can solve those problems to build a better business.

To containerize applications for a PaaS, we would need to learn basics like “What is a 12 factor application, and what’s a container?” Thinh Nguyen stepped in and gave a great description on how we use guiding principles while developing our application environment to be better for us and our customers.

Throughout all of our talks, we worked away on two pair stations very carefully brought from our lair in Cambridge. We gave away some free swag, some free candy, and raffled off some super giveaways. We thank everyone involved in preparing and executing these few days for their hard work. We also want to give a huge thanks to everyone who attended our talks (rambles) and participated in some mind-expanding conversations.

Finally, I want to close with a few notes. We always enjoy fresh perspective. If you had more to say, or you missed us during our time and you want to start a conversation, leave a comment in the comment section! If you don’t want to comment here, then drop us a line in my email. We’d love to hear from you.

Until next time, remember: Cloud Foundry, Open Source, The Way. #DellEMCDojo.

A Fresh Developer’s take on Cloud Foundry Why the third platform is the new way to iterate

Why the third platform is the new way to iterate

Coming to the Dojo a little over two months ago for my first day of work at my first (out of college) job, I was mystified by the amount of understanding and knowledge I still had to gain by using the Cloud Platform as a development and deployment environment. After the first few weeks, we started developing a brand new application for internal use. There was no existing infrastructure, but a clear idea to use a SQL database, and a web portal for the UI. I was thrilled! I worked on my fair share of web applications, so this would be a cake walk.

Great! What were the IP’s? What was the endpoint? What was the downtime for our environment on updates? What was our server technology? They said it was up to me. I began thinking of asking IT for a VM spin up our database, scripts designed to back up the data with cron jobs, registering a DNS for our website, and for a potential duplicate set for our test environment; technical aches and pains that are first steps when building a web application. I then began asking what seemed the obvious question of who to go to for the resources, and I was told we would do it all ourselves. Wait, I thought, we were supposed to deploy and maintain the VM’s set up users for the database, download the applications, and backup technology? That was easily another two or three days short term, and days of complications long term. Or so I thought.

Except…I hadn’t known that we were using Cloud Foundry, and the benefits that came from this. All of the pain points I used to face in setting up a functioning web app were moot. Instead of spending time configuring and maintaining environments, the Dojo, as a team, has put together a full-functioning web application with production environment, architecture, an extensive automated testing harness, and automated deployment. All as painlessly as we might use Github.

Removing Development Roadblocks

When I thought of the Web application, a typical structure comes to mind of the architecture for the app.

Screen Shot 2016-08-31 at 12.20.49 PM

A typical developer headache for a web application

There’s plenty of steps for our team, as developers, to maintain and care about here. But with Cloud Foundry, we can remove many of these concerns.

Lets talk about Databases first. Using service brokers as provided through CF, we can simply bind a SQL service to our application. For extra sweetness, we can use a buildpack that supports that our server is connecting to a service-provided database. The buildpack will, at runtime, connect our application’s backend into a bound SQL database, no environment variables necessary.

Servers and UI? No problem. Pushing applications to Cloud Foundry takes as many commands as pushing to git. We can also update and configure the application directly through a UI included in our environment. This means changing something we forgot to put in our three-four deployment commands doesn’t warrant another deployment. As far as firewalls go, using the proper security groups to avoid exposing secretive endpoints is a few minutes behind the scenes of a UI, or a couple commands in a terminal.

Alright, so our development roadblocks are covered. What does all this mean for our Customers?

Rapid Integration and Feedback Loops

With our Test Driven Development and Continuous Integration mantras, we can rapidly iterate on our products. With Cloud Foundry, those rapid iterations allow us to take customer feedback. Using our real product on something as quick to set up and easily accessible as CF platform is, we can respond to our customer’s concerns much more efficiently.

Allowing our customers a way to directly interact with our product in this way is not only helpful for them, but for us, for a variety of reasons. Firstly, we’re able to get valuable feedback from them in a timely fashion, not when a feature has been done and untouched for two months (less old code touching, the better). Additionally, we can test what our real runtime environment will be like for our production level product, which means faster satisfaction for customers.

Screen Shot 2016-08-31 at 12.48.14 PM

Our development cycle. Made simple with less setup behind the scenes

Finally, we don’t have to worry about maintaining servers and VMS when they may go down. Cloud Foundry can sense when an app crashes, and restart it for us. Same database, same DNS, and limited downtime. Doesn’t get much easier than that.

Final thoughts

Using Cloud Foundry also means we can simplify our development process by having a consistent API and well-documented environment surrounding our application. When we rotate around our projects, people coming into our web application are using similar knowledge and practices they might use when deploying our Minecraft Docker Containers.

It’s for the reasons in this article, and I’m sure for many more to come, that the cloud platform and Cloud Foundry has made development easier for us. It allows us to create better products, faster. We can get immediate feedback from customers and get stories accepted faster. For most developers, that’s a no brainer.

Continuous integration at the #EMCDojo CI at the EMC Dojo, Brian Roche Sr Director of Engineering

CI at the EMC Dojo, Brian Roche Sr Director of Engineering

Brian Roche

Brian Roche - Senior Director, Cloud Platform Team at Dell EMC. Brian Roche is the Leader of Dell EMC’s Cloud Platform Team. He is based in Cambridge, Massachusetts, USA at the #EMCDojo.

If you have worked on a project for the last 10 years and you have a nicely written (and probably bound) Product Requirements Document, then you’re probably working on a Waterfall project.  If you’re too busy to even ponder the first sentence because you’re overly concerned with the ’testing phase’ of the project that’s coming in a few months, then you’re definitely on a Waterfall project.

The truth is most of the IT projects in-flight today use the Waterfall methodology of Plan, Do, Check, Release.  These projects measure release cycles in months and sometimes years – not in minutes or seconds like most agile projects.  For them, releasing software is a BIG DEAL.  After all, they’ve been working on this software for 12-18 months.  These teams don’t practice the art of releasing software very often.  So what happens when we don’t build the muscle memory to do something?  We’re not very good at it. Most Waterfall teams are not very good at releasing software to their customers.  Subsequently, the product they produce reflects the fact they’re out of shape when it comes to creating installers, packages and the release process in general.

(more…)

How to Set up a Concourse Pipeline Xuebin He Dojo Developer

Xuebin He Dojo Developer

How to Set up a Concourse Pipeline

The first step to continuous integration is setting up your own CI pipeline. The #EMCdojo uses Concourse for our own pipeline and we love it! Concourse (the official CI tool for Cloud Foundry) can pull commmitted code and run tests against it, and even create a release after passing tests.

Before I tell you HOW, I’ll tell you WHY

In our workspace, our pipeline monitor is displayed on a wall right next to the team. A red box (aka failed task) is a glaring indicator that something went wrong. Usually the first person who notices shouts out “Ooh! What happened?” and then we roll up our sleeves and start debugging. Each job block can be clicked on to get output logs about what happened. The Concourse CLI lets you ‘hijack’ the container running the job for hands-on debugging. Combining these tools, it’s usually fairly quick to find a problem and fix it.

Having this automated setup, it’s easy to push small features one at a time into production and see their immediate effect on the product. We can see if the feature breaks any existing tests (unit, integration, lifecycle, etc). We also push new tests with the new feature and those are added to the pipeline. At the end of the pipeline, we know for sure if the feature is done, or still needs more work.

Step 1: Set up Concourse

Set up Server

The easiest way to set up concourse is using vagrant

You can access your concourse at 192.168.100.4:8080

Download Concourse cli

You can only start, pause, and stop pipelines or tasks on the concourse website. If you want to configure the pipeline, you have to download fly from concourse. Fly is the name of concourse cli.

Step 2: Configure Pipeline

Make a CI Folder

You can generate your CI folder under the root of your project. See code block below.

pipeline.yml will define what your pipeline looks like.

pipeline.yml

So now your pipeline should look like this:

pipeline

Using groups, we can make a different combination of jobs. Each job can have several tasks. The tasks are located here ci/tasks/*.yml.

task.yml

This defines a task. A task is like a function from inputs to outputs that succeed or fail. Each task runs in a seperate container that requires you to give the address of the docker image that you want use. You can put Dockerfile under ci/docker/. Inputs are already defined in pipeline.yml. The duplication here is to make it easy for us to run one-off tests. The outputs of the task can be reused by later tasks in the same job.

Make a Secret File

You have to generate a secret file that has all of the environments required by the pipeline. All required environments are in pipeline.yml wrapped by double curly braces.

secrets.yml

Set Pipeline

Start Pipeline

The initial state of the pipeline is paused. You have to start it by clicking the menu button on the concourse website OR with fly by:

Run One-off

You can run a one-off test for a specific job. This will not be shown in pipeline.

one-off.sh

The lines above fly execute are the environment variables, and lines below are the inputs of that task. Those are already defined in ci/tasks/*.yml.

Debug

You can hijack into the container thats running the task that you want to debug by:

If you run one-off, you can just run:

You can find build number by clicking the top right button on your pipeline page.

And you’re done! And remember: Continuous Integration = Continuous Confidence.

If you have any questions, please comment below.

Follow Us on Twitter

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.