Thriving in a world of disruption – the “Dojo Way” Brian Roche, Lead of the @DellEMCDojo

Brian Roche, Lead of the @DellEMCDojo

Brian Roche

Brian Roche - Senior Director, Cloud Platform Team at Dell EMC. Brian Roche is the Leader of Dell EMC’s Cloud Platform Team. He is based in Cambridge, Massachusetts, USA at the #EMCDojo.

We in live a world where ideas can be dreamt up, implemented and delivered more rapidly than ever before.  The pace of innovation is like nothing we’ve ever seen in our lifetime and it will likely only get faster.  Old patterns of software delivery, 12 month releases are insufficient in meeting demands of today’s marketplace.  As a result many people will face extinction if they do not change their work patterns and focus on two key objectives; customer needs and delivering software rapidly to meet that need.  Iterate and repeat.

This disruption creates opportunity if we’re in the right position to take advantage of the constant changes. In fact it is possible thrive in this new world and lay the foundation for a successful future.  At the @DellEMCDojo we have found that better way. The Dojo was created for 2 simple reasons. First to adopt a DevOps culture to achieve lower cost innovation, more rapid product delivery and to build innovative solutions that meet the needs of customers.  Second, to contribute to OS Cloud Foundry.  All of this is important because we not only demonstrate to our customers that we walk-the-walk and talk-the-talk but we also can respond to their market needs and deliver solutions to them to enable and empower them to compete effectively.

 

The dojo methodology is important but it’s what we create with this new way of working that is most important.  After all, this is a business, revenue is the score card and we measure the success of our software based on user adoption.  Here is a brief run through of the projects we’ve worked on in recent months.

 

Persistence in Cloud Foundry 

Cloud Foundry is WAY cool & 12 Factor Apps are WAY cool too BUT you and I don’t just have to worry about 12 Factor Apps.  We have legacy apps that live in our family that we still need to pay attention to. Our position is there should be a place for these legacy apps in our cool New World called Cloud Foundry. That’s why we enabled container persistence in Cloud Foundry.  Shipping in PCF 1.9 customers can take advantage of this functionality by mounting NFSv3 volumes from within their containers.  The obvious first technology integration was with Isilon; customers can now mount Isilon volumes.  As a result we have created a world where 12 Factor apps and non 12 Factor apps can live together and experience the benefit of Cloud Foundry.  We are actively developing this technology for and with customers and the response thus far has been very positive.

 

GPUaaS and Cloud Foundry 

We are seeing a growing demand for big data processing for one simple reason, the ability to make intelligent decisions about data is as important as continuous delivery.  Running those big data workloads that require massive processing power is not easy for Cloud Foundry developers today.  One obvious challenge is they cannot choose where these application workloads will be executed.  Leveraging the great work that Jack Harwood and the team in China has done on GPUaaS we were able to integrate this with Cloud Foundry.  The result is application developers will now have choice—they will be able to choose where their workloads are run leading to even greater efficiency and value from their information.

 

Blockchain and Cloud Foundry 

Blockchain technology comes up more and more when the larger Dell EMC talks to customers.  The first question is often ‘what is it and how can we leverage it?’  Up until recently we didn’t have an opinion on this technology.  So we kicked off a project with the following goals:

 

1.      Understand the different Blockchain distributions

2.      Identify ways in which we could integrate Blockchain with Cloud Foundry

3.      Make this available to the community so we can all iterate on this together

 

I’m pleased to say we now have a Blockchain implementation with Cloud Foundry and are happy to share not only the code/implementation but also our learnings.  Together as a community we can go far in developing this technology to solve real business problems.

 

 

The Dojo Effect

The Dojo team has come a long way in two years, now everyone wants to create a dojo.  The brush fires have been lit and the ‘dojo way’ is spreading.  A word of caution for those eager to get started quickly; while we’re happy with the enthusiasm and the willingness to change work patterns, there’s more to building a dojo and adopting DevOps than changing the physical space.  This methodology is nuanced, you don’t know what you don’t know especially early on.  It can be easy, especially early on in the adoption phase, to get lost and give up.  To embrace this new way of working it’s a good idea to have help from your friends, like we did from Pivotal and Pivotal Labs.  That’s where the dojo team comes in.  By working with us in a 6 week engagement we will pair with you, teaching you to fish so you form a solid foundation from which to build on after you leave.  We’re pretty strict in implementing Lean Startup and Running Lean to the letter.  Why? Because we know when you leave you’re going to relax so we want to keep the standard high knowing you may relax later.  There’s no better way to transform than to work with the @DellEMCDojo team.

Lastly, at the heart of our success is the dojo team – the people.  The incredibly talented individuals that have adopted these new work patterns and refine their art every single day by practicing at the dojo.  I would like to acknowledge and thank an amazing dojo team for riding the waves of change and finding ways to be successful regardless of what obstacles are thrown in our path.  You and your teams can find a way to thrive and enjoy the rapid pace of innovation in the same way the dojo team has.

Until next time, c ya.

Dell EMC Dojo at Hopkinton! Reviewing what we covered, and what we learned

Reviewing what we covered, and what we learned

Hey again everyone! We’re writing out today to talk about a few topics that we covered during our time at the Dell EMC Dojo Days in Hopkinton. We met with a lot of great minds and took plenty of input into how we work. We also gave a lot of insight to passersby in what we did and how we worked.

Firstly, Brain Roche and Megan Murawski debuted the famous “Transformation Talk”. We normally give this presentation in preparation for an engagement with another team, but in this case, it was given to allow open criticism and circulation of our methodology to the rest of the company. We cover in this presentation: pairing, and why it’s important to our process, why we use TDD (Test Driven Development) (and why you should too!), and our weekly meetings including Retros, IPM, and Feedback to name a few. We had plenty of great ideas and questions, as usual, and we realized twenty minutes over time that we couldn’t get Brian off the stage.

Xuebin He eventually got Brian off-stage for a talk he conducted on CICD (Continuous Integration, and Continuous Deployment). Xuebin being one of the developers at the Dojo allowed him to be a bit more technical in his talk, and cover some of the programming practices and tools we use to achieve this at the Dojo. Concourse is our tool for running our beautifully constructed tests, along with standard mocking design patterns and the code quality produced with TDD.

We picked up again on Tuesday at 1145 to talk about why a PaaS exists, and why it’s important. That talk, given by yours truly, was focused on some of the common technical roadblocks that keep developers, customers, and managers from being able to work efficiently; as well as the ways using a PaaS can solve those problems to build a better business.

To containerize applications for a PaaS, we would need to learn basics like “What is a 12 factor application, and what’s a container?” Thinh Nguyen stepped in and gave a great description on how we use guiding principles while developing our application environment to be better for us and our customers.

Throughout all of our talks, we worked away on two pair stations very carefully brought from our lair in Cambridge. We gave away some free swag, some free candy, and raffled off some super giveaways. We thank everyone involved in preparing and executing these few days for their hard work. We also want to give a huge thanks to everyone who attended our talks (rambles) and participated in some mind-expanding conversations.

Finally, I want to close with a few notes. We always enjoy fresh perspective. If you had more to say, or you missed us during our time and you want to start a conversation, leave a comment in the comment section! If you don’t want to comment here, then drop us a line in my email. We’d love to hear from you.

Until next time, remember: Cloud Foundry, Open Source, The Way. #DellEMCDojo.

THE EVENT OF THE SEASON Coming to Hopkinton December 5th and 6th

Coming to Hopkinton December 5th and 6th

Emily Kaiser

Emily Kaiser

Head of Marketing @DellEMCDojo #CloudFoundry #OpenSource #TheWay #LeanPractices #DevOps #Empathy

Before reading this blog post, I suggest that you and anyone that may be reading over your shoulder sit down. In your chairs? Great, because the excitement you are about to feel will surely leave you overwhelmed.

 

The event of the quarter is right around the corner! Please consider this your personal invitation to join us for DOJO DAYS next Monday and Tuesday, December 5th and 6th from 10 am to 4 pm in the 176 Café out at Dell EMC in Hopkinton.

official-dojo-days-logo

DevOps is a hot word in the world of technology. It represents a way of working and technique that aims at making the customer omnipresent in everything that is created and delivered. Companies today consider this way the future.

For this reason, Dell EMC and Pivotal paired to create the first ever Dell EMC – Pivotal Cloud Foundry Dojo. Our goals are centered around personal, team and company transformation. Through the practice of modern software development, we spend our days contributing to Cloud Foundry Foundation sanctioned Open Source projects through R&D modernization and methodology known as ‘the way’ (XP, Lean Startup).

Many know us as the team in Cambridge with great food and beverage (no really, it’s amazing- come visit us to see for yourself). We realize, though, that the commute isn’t ideal for most. So, we bring you Dojo Days to experience #adayinthelifeofthedojo, with a replication of our office environment and our team in action representing the steps we took to transform.

In addition to seeing #adayinthelifeofthedojo, you will have the chance to pair with us, join our team for lightning sessions that are driven by the audience’s goals (schedule below) and have the chance to win a Dell EMC Dojo branded Patagonia, S’well water bottles, t-shirts, and more!

 

The schedule of the lightning sessions are as follows. Please come by the Café day of to sign up and/or just drop in:

Monday, December 5th:

11:45 am to 12:45 pm: Steps to Transformation starring Brian Roche and Megan Murawski

2:00 pm to 2:30 pm: TDD and CI/CD starring Xuebin He

 

Tuesday, December 6th:

11:45 am to 12:45 pm: PaaS, and Why your Developers Care starring Gary White

1:30 pm to 2:00 pm: Factor, Cloud Native, and Containers (oh my!) starring Thinh Nguyen

 

If you have any questions, please contact Emily Kaiser at Emily.Kaiser@dell.com. Otherwise, we cannot wait to see you, your team, and all of your friends out in Hopkinton next week!

Getting Started with Cloud Foundry Volume Services Trying out Persistent File Systems in Your Cloud Foundry Applications with PCFDev

Trying out Persistent File Systems in Your Cloud Foundry Applications with PCFDev

This week, the PCFDev team released a new version of PCFDev that includes local-volume services out of the box.  This new release of PCFDev gives us an option to get up and running with volume services that is an order of magnitude easier than any of the options we had before. We thought it would be a good time to write a post detailing the steps to try out volume services with your Cloud Foundry applications, now that the post doesn’t need to be quite so long.

1. Install PCFDev

(Time: 30-90 minutes, depending on your internet connection speed, and what you already have installed.)

Instructions for PCFDev installation can be found here.  Installation requires VirtualBox and the Cloud Foundry CLI, and you will need to have (or create) an account on Pivotal Network.  Don’t be deterred.  It’s free and easy.  If you already have PCFDev, make sure that you have version 0.22.0 or later.

Once you have installed PCFDev and successfully started it up, you should see something like this:

screen-shot-2016-11-16-at-4-39-13-pm

Now log in and select the pcfdev-org space:

2. Create a New Volume

(Time: 1 minute)

If you have the right version of PCFDev, then the local-volume service broker should already be installed.  You can tell this by typing

You should see local-volume in the list of available services:

screen-shot-2016-11-16-at-4-43-59-pm

Use cf create-service to create a new file volume that you can bind to your application:

3. Push an Application

(Time: 5 minutes)

For the purposes of this blog post, we’ll use the “pora” test application that we use in our persi CI pipeline. “Pora” is the persistent version of the Cloud Foundry acceptance test “Dora” application.  It is a simple ‘hello world” app that writes a message into a file, and then reads it back out again.

To get the Pora app, first clone the persi-acceptance-tests github repo:

Now change to the pora folder, and push the app to cf with the “no-start” option:

4. Bind the Service to the Application and Start the App

(Time: 5 minutes)

The cf “bind-service” command makes our new volume available to the Pora application.

Now start the pora application:

Once the app is started, you can use curl with the reported url to make sure it is reachable.  The default endpoint for the application will just report back the instance index for the application:

To test the persistence feature, add “/write” to the end of the url.  This will cause the pora app to pull the location of the shared volume from the VCAP_SERVICES environment variable, create a file, write a message into it, and then read the message back out:

5. Use Persistent Volumes with Your Own Application

By default, the volume service broker will generate a container mount path for your mounted volume, and it will pass that path into your application via the VCAP_SERVICES environment variable. You can see this when you type “cf env pora” which produces output like this:

In your application, you can parse the VCAP_SERVICES environment variable as json and determine the value of “container_path” to find the location of your shared volume in the container, or simply parse out the path with a regular expression.  You can see an example of this code flow in the Pora application here.

If it isn’t practical to connect to an arbitrary location in your application for some reason, (e.g. the directory is hard coded everywhere, or your code is precompiled, or you are reluctant to change it) then you can also tell the broker to put the volume in a specific location when you bind the application to the service.  To do that, unbind the service, and then rebind it using the -c option.  This will create any necessary directories on the container, and put the volume mount in the specific path you need:

Type “cf restage pora” to restart the application in this new configuration, and you should observe that the application operates as before, except that it now accesses the shared volume from “/my/specific/path”.  You can also see the new path transmitted to the application by typing “cf env pora”.

6. If You Need to Copy Data

If you need to copy existing data into your new volume in order for your application to use it, you will need to get the data onto the PCFDev host, and then copy it into the source directory that the local-volume service uses to mimic a shared volume.  In a real-world scenario, these steps wouldn’t be necessary because the volume would come from a real shared filesystem that you could mount and copy content to, but the local-volume service is a test service that mimics shared volumes for simple cloudfoundry deployments.

The key that PCFDev uses for scp/ssh is located in ~/.pcfdev/vms/key.pem.  This file must be tweaked to reduce its permissions before it can be used by scp:

Now invoke scp to copy your content across to the PCFDev VM.  The example below copies a file named “assets.tar” to the var/vcap/data directory on the VM:

Now, use “cf dev ssh” to open a ssh session into the vm, and cd to /var/vcap/data.  You should find your file there:

“ls volumes/local/_volumes” gives us a listing of the volumes that have been created.  If you are following this tutorial, there should be only one directory in there, with a long ugly name.  Move the file into that directory, and then exit the ssh session

Finally, invoke “cf ssh pora” to open a ssh session into the cloudfoundry app.  This will allow you to verify that the data is available to your application.  If you used the -c option to specify a mount path, you should find your content there.  If not, you will find it under “/var/vcap/data/{some-guid}”:

7. Where to Find More Information

The best place to reach the CF Diego Persistence team is on our Slack channel here: https://cloudfoundry.slack.com/messages/persi/. We’re happy to hear from you with any questions you have, and to help steer you in the right direction.

If you don’t already have an account on CF slack, you can create an account here: https://cloudfoundry.slack.com/signup

For an introduction to Volume Services in Cloud Foundry, refer to this document: http://bit.ly/2gHPVJM

Or watch our various presentations this year:

CF Summit Frankfurt 2016: https://www.youtube.com/watch?v=zrQPns47xho

Spring One Platform 2016: https://www.youtube.com/watch?v=VisP5ebZoWw

CF Summit Santa Clara 2016: https://www.youtube.com/watch?v=ajNoPi1uMjQ

Bringing Cloud Foundry to You… Exciting Upcoming Events that the Cool Kids are Attending

Exciting Upcoming Events that the Cool Kids are Attending

Emily Kaiser

Emily Kaiser

Head of Marketing @DellEMCDojo #CloudFoundry #OpenSource #TheWay #LeanPractices #DevOps #Empathy

For those of you who follow us on Twitter (@DellEMCDojo for those that don’t) or via this blog, I would guess it is pretty evident that we just love Cloud Foundry. We take pride in being a member of the foundation mostly because it is with this platform that we feel we are delivering and practicing what we preach; living out a true DevOps culture most conducive to productivity and quality product and delivery.

We are lucky enough to just be returning from Cloud Foundry Summit where upwards of 800 developers and other market adopters joined in Frankfurt, Germany to learn from other community members and to be a part of the PaaS’s business direction. With the Native Hybrid Cloud team, we at the Dojo spent our days representing DellEMC through presentations and booth shifts. We left feeling even more confident in our daily choice to use Cloud Foundry and to further evangelize the PaaS for the betterment of the larger technical OS community.

It is because of this that we are excited to announce that we will be hosting and supporting three Cloud Foundry Days here in the near future. Cloud Foundry Days are new, one-day educational events aimed at developers and operators who want to connect with businesses, learn about the platform, and share knowledge. The first that we will be hosting with the support of TIBCO, Pivotal, Telstra, Stark & Wayne, among others, is going to be taking place on October 18th in Sydney, Australia. If you have any interest in attending or know of others who may be, please find the registration link here: https://www.eventbrite.com.au/e/cloud-foundry-day-sydney-australia-tickets-27935602138. The second of these events will be taking place on October 20th in Melbourne, Australia and will also be supported by TIBCO, Pivotal, Telstra, Stark & Wayne, among others. The registration link for this event can be found here: https://www.eventbrite.com.au/e/cloud-foundry-day-melbourne-australia-tickets-27985249635. Last, but definitely not least, is the CF Day in Shanghai, China on October 25th where we will be supporting GE (Registration information to be available within the next day or two. Please contact us via twitter if you are interested).

We truly hope that these events will be fruitful in their reach and effectiveness, and help our technical community discern the best option for themselves and the businesses they affect. Teamwork makes the dream work!

NFS app support in under 5 minutes with CloudFoundry, Wahooo!

Peter Blum

Developers want quick and easy access to storage! They don’t care how they get it, they just want it now. Let me show you how a developer can go from 0 to 100 with persistence in CloudFoundry, starting with Dell EMC Isilon.

Cloud Foundry. Open Source. The Way. Dell EMC [⛩] Dojo.

Oh look! It’s a Docker Container Running on CloudFoundry!

Peter Blum

Not many people know how simple it is to run and scale there containerized application on CloudFoundry. Checkout this simple and cool video showcasing how it can be done!

Cloud Foundry. Open Source. The Way. Dell EMC [⛩] Dojo.

 

A Fresh Developer’s take on Cloud Foundry Why the third platform is the new way to iterate

Why the third platform is the new way to iterate

Coming to the Dojo a little over two months ago for my first day of work at my first (out of college) job, I was mystified by the amount of understanding and knowledge I still had to gain by using the Cloud Platform as a development and deployment environment. After the first few weeks, we started developing a brand new application for internal use. There was no existing infrastructure, but a clear idea to use a SQL database, and a web portal for the UI. I was thrilled! I worked on my fair share of web applications, so this would be a cake walk.

Great! What were the IP’s? What was the endpoint? What was the downtime for our environment on updates? What was our server technology? They said it was up to me. I began thinking of asking IT for a VM spin up our database, scripts designed to back up the data with cron jobs, registering a DNS for our website, and for a potential duplicate set for our test environment; technical aches and pains that are first steps when building a web application. I then began asking what seemed the obvious question of who to go to for the resources, and I was told we would do it all ourselves. Wait, I thought, we were supposed to deploy and maintain the VM’s set up users for the database, download the applications, and backup technology? That was easily another two or three days short term, and days of complications long term. Or so I thought.

Except…I hadn’t known that we were using Cloud Foundry, and the benefits that came from this. All of the pain points I used to face in setting up a functioning web app were moot. Instead of spending time configuring and maintaining environments, the Dojo, as a team, has put together a full-functioning web application with production environment, architecture, an extensive automated testing harness, and automated deployment. All as painlessly as we might use Github.

Removing Development Roadblocks

When I thought of the Web application, a typical structure comes to mind of the architecture for the app.

Screen Shot 2016-08-31 at 12.20.49 PM

A typical developer headache for a web application

There’s plenty of steps for our team, as developers, to maintain and care about here. But with Cloud Foundry, we can remove many of these concerns.

Lets talk about Databases first. Using service brokers as provided through CF, we can simply bind a SQL service to our application. For extra sweetness, we can use a buildpack that supports that our server is connecting to a service-provided database. The buildpack will, at runtime, connect our application’s backend into a bound SQL database, no environment variables necessary.

Servers and UI? No problem. Pushing applications to Cloud Foundry takes as many commands as pushing to git. We can also update and configure the application directly through a UI included in our environment. This means changing something we forgot to put in our three-four deployment commands doesn’t warrant another deployment. As far as firewalls go, using the proper security groups to avoid exposing secretive endpoints is a few minutes behind the scenes of a UI, or a couple commands in a terminal.

Alright, so our development roadblocks are covered. What does all this mean for our Customers?

Rapid Integration and Feedback Loops

With our Test Driven Development and Continuous Integration mantras, we can rapidly iterate on our products. With Cloud Foundry, those rapid iterations allow us to take customer feedback. Using our real product on something as quick to set up and easily accessible as CF platform is, we can respond to our customer’s concerns much more efficiently.

Allowing our customers a way to directly interact with our product in this way is not only helpful for them, but for us, for a variety of reasons. Firstly, we’re able to get valuable feedback from them in a timely fashion, not when a feature has been done and untouched for two months (less old code touching, the better). Additionally, we can test what our real runtime environment will be like for our production level product, which means faster satisfaction for customers.

Screen Shot 2016-08-31 at 12.48.14 PM

Our development cycle. Made simple with less setup behind the scenes

Finally, we don’t have to worry about maintaining servers and VMS when they may go down. Cloud Foundry can sense when an app crashes, and restart it for us. Same database, same DNS, and limited downtime. Doesn’t get much easier than that.

Final thoughts

Using Cloud Foundry also means we can simplify our development process by having a consistent API and well-documented environment surrounding our application. When we rotate around our projects, people coming into our web application are using similar knowledge and practices they might use when deploying our Minecraft Docker Containers.

It’s for the reasons in this article, and I’m sure for many more to come, that the cloud platform and Cloud Foundry has made development easier for us. It allows us to create better products, faster. We can get immediate feedback from customers and get stories accepted faster. For most developers, that’s a no brainer.

Multi-Cloud MineCraft! Minecraft + Docker + CloudFoundry + vSphere + RackHD + ScaleIO = Awesomeness

Minecraft + Docker + CloudFoundry + vSphere + RackHD + ScaleIO = Awesomeness

Peter Blum

If you didn’t know, VMworld begins next week! Here at the Dojo we thought we would show you a cool project we’ve been working on. MINECRAFT in the cloud!!

With all the clouds out there, both public and private, we wanted to bridge the gap between them. We took a public vSphere cloud and a private bare metal cloud, and layered CloudFoundry ontop. Then with a simple CF push we spin up a Minecraft server from a Docker image. Hmm don’t know if I could put more buzz words into a sentence if I tried…anyhow check it out!

Big shout out to VMWare, RackHD, RexRay, Libstorage, and of course CloudFoundry for helping us make it possible. Also here is a link to our Github (https://github.com/EMC-Dojo/CloudFoundry-MineCraft) where you can find the Docker Image that you can run inside cloudFoundry for your own private Minecraft server.

Cloud Foundry. Open Source. The Way. EMC [⛩] Dojo.

Using Docker Container in Cloud Foundry

Using Docker Container in Cloud Foundry

As we all know, we can push source code to CF directly, and CF will compile it and create a container to run our application. Life is so great with CF.

But sometimes, for some reason, such as our App needs a special setup or we want to run an app on different platforms or infrastructures, we may already have a preconfigured container for our App. This won’t block our way to CF at all. This post will show you how to push docker images to CF.

Enable docker feature for CF

We can turn on docker support with the following cf command

We can also turn it off by

Push docker image to CF

Unlike the normal way, CF won’t try to build our code and run it inside the image we specified. CF would assume that you already put  everything you need into your docker image. We have to rebuild the docker image every time we push a change to our repository.

We also need to tell CF how to start our app inside the image by specifying the start command. We can either put it as an argument for cf push or put it into manifest.yml as below.

In this example, we are using an official docker image from docker hub. In the start command, we clone our demo repo from Github, do something and run our code.

Update Diego with private docker registry

If you are in the EMC network, you may not able to use Docker Hub due to certificate issues. In this case, you need to setup a private docker registry. The version of registry needs to be V2 for now. Also, you have to redeploy your CF or Diego with the changes being shown below.

Replace 12.34.56.78:9000 with your own docker registry ip and port.

Then, you need to create a security group to reach your private docker registry. You can put the definition of this security group into docker.json as shown below

And run

Now you can re-push to CF by

Follow Us on Twitter

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.