Author Archive

Julian Hjortshoj

Julian Hjortshoj

Julian is a 12 year veteran of DellEmc, and the current PM of the Cloud Foundry Diego Persistence team.

Running Legacy Apps on CloudFoundry with NFS How to re-platform your apps and connect to existing shared volumes using CloudFoundry Volume Services

How to re-platform your apps and connect to existing shared volumes using CloudFoundry Volume Services

This week the Cloud Foundry Diego Persistence team released the 1.0 version of our nfs-volume-release for existing NFS data volumes.  This Bosh release provides the service broker and volume driver components necessary to quickly connect Cloud Foundry deployed applications to existing NFS file shares.

In this post, we will take a look at the steps required to add the nfs-volume-release to your existing Cloud Foundry deployment, and the steps required after that to get your existing file system based application moved to Cloud Foundry.

Deploying nfs-volume-release to Cloud Foundry

If you are using OSS cloud foundry, you’ll need to deploy the service broker and driver into your cloudfoundry deployment.  To do this, you will need to colocate the nfsv3driver on the Diego cells in your Cloud Foundry deployment, and then run the nfs service broker either as a Cloud Foundry application or a BOSH deployment.

Detailed instructions for deploying the driver are here.

Detailed instructions for deploying the broker are here.

If you are using PCF, nfs-volume-release is built in.  As of PCF 1.10, you can deploy the broker and driver through simple checkbox configuration in the Advanced features tab in Ops Manager.  Details here.

Moving your application into Cloud Foundry

There are a range of issues you might hit when moving a legacy application from a single server context into Cloud Foundry, and most of them are outside the scope of this article.  See the last section of this article for a good reference discussing how to migrate more complex applications.  For the purposes of this article we’ll focus on a relatively simple content application that’s already well suited to run in CF except that it requires a file system.  We’ll use servocoder/RichFileManager as our example application.  It supports a couple different HTTP backends, but we’ll use the PHP backend in this example.

Once you have cloned the RichFileManager repository and followed the set up instructions, you should theoretically be able to run the application in Cloud Foundry’s php buildpack with a simple cf push from the RichFileManager root directory:

But RichFileManager requires the gd package which isn’t included by default in the php buildpack.  If we push the application as-is, file upload operations will fail after RichFileManager dies while trying to create thumbnail images for uploaded files.  To fix this, we need to create a .bp-options directory in the root folder for our application and put a file named options.json in it with the following content:

Re-pushing the application fixes the problem.  Now we are able to upload files and use all the features of RichFileManager:

But we aren’t done yet! By default, the RichFileManager application stores uploaded file content in a subdirectory of the application itself.  As a result, any file data will be treated as ephemeral by cloudfoundry and discarded when the application restarts.  To see why this is a problem, upload some files, and then type:

When you refresh the application in your browser, you’ll see that your uploaded files are gone!  That’s why you need to bind a volume service to your application.

In order to do that, we first need to tweak the application a little to tell it that we want to put files in an external folder.  Inside the application, open connectors/php/config.php in your editor of choice, and change the value for “serverRoot” to false.  Also set the value of “fileRoot” to “/var/vcap/data/content”.  (As of today, cloudfoundry has the limitation that volume services cannot create new root level folders in the container.  Soon that limitation will be lifted, but in the mean time, /var/vcap/data is a safe place to bind our storage directory to.)

Now push the application again:

When you go back to the application, you should see that it is completely broken and hangs waiting to get content.  That’s because we told it to use a directory that doesn’t yet exist.  To fix that, we need to create a volume service, and bind it to our application.  You can follow the instructions on the nfs-volume-release to set up an nfs test server in your environment, or if you already have an NFS server available (for example, Isilon, ECS, Netapp or the like) you can skip the setup steps and go directly to the service broker registration step.  Once you have created a volume service instance, bind that service to your application:

If you are using an existing NFS server, you will likely need to specify different values for uid and gid.  Pick values that correspond to a user with write access to the share you’re using.

Now restage the application:

You should see that the application now works properly again.  Furthermore, you can now “cf restart” your application, and “cf scale” it to run multiple instances, and it will continue to work and to serve up the same files.

Caveats

Volume services enable filesystem based applications to overcome a major barrier to cloud deployment, but they will not enable all applications to run seamlessly in the cloud.  Applications that rely on transactions across http requests, or otherwise store state in memory will still fail to run properly when scaled out to more than one instance in cloud foundry.  CF provides best-effort session stickiness for any application that sets a JSESSIONID cookie, but no guarantees that traffic will not get routed to another instance.

More detail on steps to make complex applications run in the cloud can be found in this article.

Getting Started with Cloud Foundry Volume Services Trying out Persistent File Systems in Your Cloud Foundry Applications with PCFDev

Trying out Persistent File Systems in Your Cloud Foundry Applications with PCFDev

This week, the PCFDev team released a new version of PCFDev that includes local-volume services out of the box.  This new release of PCFDev gives us an option to get up and running with volume services that is an order of magnitude easier than any of the options we had before. We thought it would be a good time to write a post detailing the steps to try out volume services with your Cloud Foundry applications, now that the post doesn’t need to be quite so long.

1. Install PCFDev

(Time: 30-90 minutes, depending on your internet connection speed, and what you already have installed.)

Instructions for PCFDev installation can be found here.  Installation requires VirtualBox and the Cloud Foundry CLI, and you will need to have (or create) an account on Pivotal Network.  Don’t be deterred.  It’s free and easy.  If you already have PCFDev, make sure that you have version 0.22.0 or later.

Once you have installed PCFDev and successfully started it up, you should see something like this:

screen-shot-2016-11-16-at-4-39-13-pm

Now log in and select the pcfdev-org space:

2. Create a New Volume

(Time: 1 minute)

If you have the right version of PCFDev, then the local-volume service broker should already be installed.  You can tell this by typing

You should see local-volume in the list of available services:

screen-shot-2016-11-16-at-4-43-59-pm

Use cf create-service to create a new file volume that you can bind to your application:

3. Push an Application

(Time: 5 minutes)

For the purposes of this blog post, we’ll use the “pora” test application that we use in our persi CI pipeline. “Pora” is the persistent version of the Cloud Foundry acceptance test “Dora” application.  It is a simple ‘hello world” app that writes a message into a file, and then reads it back out again.

To get the Pora app, first clone the persi-acceptance-tests github repo:

Now change to the pora folder, and push the app to cf with the “no-start” option:

4. Bind the Service to the Application and Start the App

(Time: 5 minutes)

The cf “bind-service” command makes our new volume available to the Pora application.

Now start the pora application:

Once the app is started, you can use curl with the reported url to make sure it is reachable.  The default endpoint for the application will just report back the instance index for the application:

To test the persistence feature, add “/write” to the end of the url.  This will cause the pora app to pull the location of the shared volume from the VCAP_SERVICES environment variable, create a file, write a message into it, and then read the message back out:

5. Use Persistent Volumes with Your Own Application

By default, the volume service broker will generate a container mount path for your mounted volume, and it will pass that path into your application via the VCAP_SERVICES environment variable. You can see this when you type “cf env pora” which produces output like this:

In your application, you can parse the VCAP_SERVICES environment variable as json and determine the value of “container_path” to find the location of your shared volume in the container, or simply parse out the path with a regular expression.  You can see an example of this code flow in the Pora application here.

If it isn’t practical to connect to an arbitrary location in your application for some reason, (e.g. the directory is hard coded everywhere, or your code is precompiled, or you are reluctant to change it) then you can also tell the broker to put the volume in a specific location when you bind the application to the service.  To do that, unbind the service, and then rebind it using the -c option.  This will create any necessary directories on the container, and put the volume mount in the specific path you need:

Type “cf restage pora” to restart the application in this new configuration, and you should observe that the application operates as before, except that it now accesses the shared volume from “/my/specific/path”.  You can also see the new path transmitted to the application by typing “cf env pora”.

6. If You Need to Copy Data

If you need to copy existing data into your new volume in order for your application to use it, you will need to get the data onto the PCFDev host, and then copy it into the source directory that the local-volume service uses to mimic a shared volume.  In a real-world scenario, these steps wouldn’t be necessary because the volume would come from a real shared filesystem that you could mount and copy content to, but the local-volume service is a test service that mimics shared volumes for simple cloudfoundry deployments.

The key that PCFDev uses for scp/ssh is located in ~/.pcfdev/vms/key.pem.  This file must be tweaked to reduce its permissions before it can be used by scp:

Now invoke scp to copy your content across to the PCFDev VM.  The example below copies a file named “assets.tar” to the var/vcap/data directory on the VM:

Now, use “cf dev ssh” to open a ssh session into the vm, and cd to /var/vcap/data.  You should find your file there:

“ls volumes/local/_volumes” gives us a listing of the volumes that have been created.  If you are following this tutorial, there should be only one directory in there, with a long ugly name.  Move the file into that directory, and then exit the ssh session

Finally, invoke “cf ssh pora” to open a ssh session into the cloudfoundry app.  This will allow you to verify that the data is available to your application.  If you used the -c option to specify a mount path, you should find your content there.  If not, you will find it under “/var/vcap/data/{some-guid}”:

7. Where to Find More Information

The best place to reach the CF Diego Persistence team is on our Slack channel here: https://cloudfoundry.slack.com/messages/persi/. We’re happy to hear from you with any questions you have, and to help steer you in the right direction.

If you don’t already have an account on CF slack, you can create an account here: https://cloudfoundry.slack.com/signup

For an introduction to Volume Services in Cloud Foundry, refer to this document: http://bit.ly/2gHPVJM

Or watch our various presentations this year:

CF Summit Frankfurt 2016: https://www.youtube.com/watch?v=zrQPns47xho

Spring One Platform 2016: https://www.youtube.com/watch?v=VisP5ebZoWw

CF Summit Santa Clara 2016: https://www.youtube.com/watch?v=ajNoPi1uMjQ

Building a healthy Concourse CI pipeline for Bosh deployed products

The Cloudfoundry Diego Persistence team recently spent a fair amount of time and effort building and refactoring the CI pipeline for our Ceph filesystem, volume driver, and service broker.  The end state from this exercise, while not perfect, is nonetheless pretty darn good:  It deploys Cloudfoundry, Diego, and a Cephfs cluster, along with our volume driver and service broker.  It runs our code through unit tests, certification tests, and acceptance tests.  It keeps our deployment up to date with the latest releases of CloudFoundry and the latest development branch changes to Diego.  It does all of this with minimal rework or delay; changes in our driver/broker bosh release typically flow through the pipeline in about 10 minutes.  

But our first attempt at creating the pipeline did not work very well or very quickly, so we thought it would be worth documenting our initial assumptions, what was wrong about them, some of what we learned while fixing them.

Our First Stab at It

We started with a set of assumptions about what we could run quickly and what would run slowly, and we tried to organize our pipeline around those assumptions to make sure that the quick stuff didn’t get blocked by the slow stuff.

Assumptions:

  • Cephfs cluster deployment is slow–it requires us to apt-get a largish list of parts and then provision a cluster.  This can take 20-30 minutes.
  • Since cluster deployment is slow, and we share a bosh release for the cephfs bosh job and our driver and broker bosh jobs, we should only trigger cephfs deployment nightly when nobody is waiting–we shouldn’t trigger it when our bosh release is updated.
  • Redeploying Cephfs is not safe–to make sure that it stays in a clean state, we should undeploy it before deploying it again.
  • CloudFoundry deployment is slow–we should not automatically pick up new CF releases because it might paralyze our pipeline during the work day.
  • The pipeline should clean up on failure–bad deployments of cephfs should get torn down automatically.

 

What We Eventually Learned

Our first pass at the pipeline (mostly) worked, but it was slow and inefficient.  Because we structured it to deploy some of the critical components nightly or on demand, and we tore down the ceph file system vm before redeploying it, any time we needed an update, we had to wait a long time.  In the case of cephfs, we also had to create a shadow pipeline just for manually triggering cephfs redeployment.  It turned out that most of of the assumptions above were wrong, so let’s take another look at those:

 

Bad Assumptions:

  • Cephfs cluster deployment is slow. This is only partially true.  Because we installed cephfs using apt-get, we were doing an end-run around Bosh package management, effectively ensuring that we would re-do work in our install script whether it was necessary or not.  We switched from apt-get to Bosh managed debian packages and that sped things up a lot.  Bosh caches packages and only fetches things that have actually changed.
  • We should only trigger cephfs deployment nightly or we will repeat slow cephfs deployments whenever code changes.  This is totally untrue.  Bosh is designed to detect changes from one version to the next, so when the broker job or the driver job changes, but cephfs hasn’t changed, deploying the cephfs job will result in a no-op for bosh.  
  • Redeploying Cephfs is not safe.  This might be partially true. In theory our ceph filesystem could get corrupted in ways that would cause the pipeline to keep failing, but treating this operation as unsafe is somewhat antithetical to cloud operations.  Bosh jobs should as much as possible be safe to redeploy without removing them.
  • CloudFoundry deployment is slow.  This is usually not true.  When there are new releases of CloudFoundry, they deploy incrementally just like other bosh deployments, so only the changed jobs will result in deployment changes.  The real culprit in slow deployment times happens when there is an update to the bosh stemcell, and bosh needs to download the stemcell before it can deploy.  In order to keep that from slowing down our pipeline during the workday, we created a “nightly stemcell” task in the pipeline that doesn’t do anything, but can only run at night.  Using the latest passed stemcell from that task, and setting the stemcell as a trigger in our deploy tasks ensures that when there is a stemcell change, our pipeline will pick it up at night, and redeploy with it, and that we will never have to wait for a stemcell download during the day:

  • The pipeline should clean up on failureThis is generally a bad practice.  It means that we have no way of diagnosing failures in the pipeline.  Teardown after failure also doesn’t restore the health of the pipeline, unless the deployments in question are re-deployed after, but in the case of a deployment error, that could easily result in a tight loop of deployment and undeployment, so we never did that.

Where We Ended Up

Screen Shot 2016-06-29 at 2.57.33 PM

After we corrected all of our wrong assumptions, our pipeline is in much better shape:

  • Bosh deployments are incremental and frequent.  We pick up new releases as soon as they happen, and we re-test against them, so we get early warning of failures even when we didn’t make the breaking changes.
  • Our bosh job install scripts are as much as possible idempotent.  The only undeploy jobs we have in the pipeline are manually triggered.
  • We trigger slow stemcell downloads at night when nobody is working, and stick to the same stemcells during the day to avoid slow downloads.
  • Since we share the same bosh release for 3 different deployments (broker, driver, and file system) we trigger deployment of all 3 things whenever our bosh release changes.  Since Bosh is clever about not doing anything for unchanged jobs, this is a much easier approach than trying to manage separate versions of the bosh release for different jobs.
  • We use concourse serial groups to force serialization between the tasks that deploy things and the tasks that rely on those deployments.  Serial groups are far from perfect–they operate as a simple mutex with no read/write lock semantics–but for our purposes they proved to be good enough, and they are far easier than implementing our own locks.

The yaml for our current pipeline is here for reference.

Housekeeping

In addition to our nightly job to download stemcells, we also run a nightly task to clean up bosh releases by invoking bosh cleanup.  This is a very good idea–otherwise bosh keeps everything that’s been uploaded to it, which can quickly use up available disk space.

At some point in the future, we will probably want to add additional tasks to the pipeline to clean out our Amazon S3 buckets, but so far we haven’t done that.

Thanks

A special thanks to Connor Braa who recently joined our team from the Diego team where he did a great deal of Concourse wrangling.  Connor is responsible for providing us with most of the insights in this post.

Follow Us on Twitter

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.