Author Archive

Julian Hjortshoj

Julian Hjortshoj

Julian is a 12 year veteran of DellEmc, and the current PM of the Cloud Foundry Diego Persistence team.

Getting Started with Cloud Foundry Volume Services Trying out Persistent File Systems in Your Cloud Foundry Applications with PCFDev

Trying out Persistent File Systems in Your Cloud Foundry Applications with PCFDev

This week, the PCFDev team released a new version of PCFDev that includes local-volume services out of the box.  This new release of PCFDev gives us an option to get up and running with volume services that is an order of magnitude easier than any of the options we had before. We thought it would be a good time to write a post detailing the steps to try out volume services with your Cloud Foundry applications, now that the post doesn’t need to be quite so long.

1. Install PCFDev

(Time: 30-90 minutes, depending on your internet connection speed, and what you already have installed.)

Instructions for PCFDev installation can be found here.  Installation requires VirtualBox and the Cloud Foundry CLI, and you will need to have (or create) an account on Pivotal Network.  Don’t be deterred.  It’s free and easy.  If you already have PCFDev, make sure that you have version 0.22.0 or later.

Once you have installed PCFDev and successfully started it up, you should see something like this:


Now log in and select the pcfdev-org space:

○ → cf login -a --skip-ssl-validation
API endpoint:

Email> admin


Select an org (or press enter to skip):
1. pcfdev-org
2. system

Org> 1
Targeted org pcfdev-org

Targeted space pcfdev-space

API endpoint: (API version: 2.58.0)
User: admin
Org: pcfdev-org
Space: pcfdev-space

○ →

2. Create a New Volume

(Time: 1 minute)

If you have the right version of PCFDev, then the local-volume service broker should already be installed.  You can tell this by typing

cf marketplace

You should see local-volume in the list of available services:


Use cf create-service to create a new file volume that you can bind to your application:

○ → cf create-service local-volume free-local-disk myvolume
Creating service instance myvolume in org pcfdev-org / space pcfdev-space as admin...

○ →

3. Push an Application

(Time: 5 minutes)

For the purposes of this blog post, we’ll use the “pora” test application that we use in our persi CI pipeline. “Pora” is the persistent version of the Cloud Foundry acceptance test “Dora” application.  It is a simple ‘hello world” app that writes a message into a file, and then reads it back out again.

To get the Pora app, first clone the persi-acceptance-tests github repo:

○ → git clone
Cloning into 'persi-acceptance-tests'...
remote: Counting objects: 228, done.
remote: Compressing objects: 100% (97/97), done.
remote: Total 228 (delta 120), reused 222 (delta 114), pack-reused 0
Receiving objects: 100% (228/228), 39.42 KiB | 0 bytes/s, done.
Resolving deltas: 100% (120/120), done.
Checking connectivity... done.

○ → 

Now change to the pora folder, and push the app to cf with the “no-start” option:

○ → cd persi-acceptance-tests/assets/pora

○ → cf push pora --no-start
Using manifest file /Users/pivotal/workspace/eraseme/persi-acceptance-tests/assets/pora/manifest.yml

Creating app pora in org pcfdev-org / space pcfdev-space as admin...

Creating route

Binding to pora...

Uploading pora...
Uploading app files from: /Users/pivotal/workspace/eraseme/persi-acceptance-tests/assets/pora
Uploading 1K, 2 files
Done uploading
○ →

4. Bind the Service to the Application and Start the App

(Time: 5 minutes)

The cf “bind-service” command makes our new volume available to the Pora application.

○ → cf bind-service pora myvolume
Binding service myvolume to app pora in org pcfdev-org / space pcfdev-space as admin...
TIP: Use 'cf restage pora' to ensure your env variable changes take effect

○ →

Now start the pora application:

○ → cf start pora
Starting app pora in org pcfdev-org / space pcfdev-space as admin...
Downloading go_buildpack...
Downloaded go_buildpack (320.8M)
Creating container
Successfully created container
Downloading app package...
Downloaded app package (1.1K)
-------> Buildpack version 1.7.10
-----> Installing go1.6.3... done
Downloaded [file:///tmp/buildpacks/a62bb8a9fc6ecb01a06f86c020c7a142/dependencies/https___storage.googleapis.com_golang_go1.6.3.linux-amd64.tar.gz]
-----> Running: go install -v -tags cloudfoundry .
Exit status 0
Staging complete
Uploading droplet, build artifacts cache...
Uploading build artifacts cache...
Uploading droplet...
Uploaded build artifacts cache (81.6M)
Uploaded droplet (2.3M)
Uploading complete
Destroying container
Successfully destroyed container

1 of 1 instances running

App started


App pora was started using this command `pora`

Showing health and status for app pora in org pcfdev-org / space pcfdev-space as admin...

requested state: started
instances: 1/1
usage: 256M x 1 instances
last uploaded: Thu Nov 17 03:20:29 UTC 2016
stack: cflinuxfs2
buildpack: go_buildpack

state since cpu memory disk details
#0 running 2016-11-16 07:38:32 PM 0.0% 0 of 256M 0 of 512M

○ →

Once the app is started, you can use curl with the reported url to make sure it is reachable.  The default endpoint for the application will just report back the instance index for the application:

○ → curl
instance index: 0
○ →

To test the persistence feature, add “/write” to the end of the url.  This will cause the pora app to pull the location of the shared volume from the VCAP_SERVICES environment variable, create a file, write a message into it, and then read the message back out:

○ → curl
Hello Persistent World!
○ →

5. Use Persistent Volumes with Your Own Application

By default, the volume service broker will generate a container mount path for your mounted volume, and it will pass that path into your application via the VCAP_SERVICES environment variable. You can see this when you type “cf env pora” which produces output like this:

  "local-volume": [
    "volume_mounts": [
      "container_path": "/var/vcap/data/0ce3b464-3d4c-45af-9e78-29990d7ddac1",
      "mode": "rw"

In your application, you can parse the VCAP_SERVICES environment variable as json and determine the value of “container_path” to find the location of your shared volume in the container, or simply parse out the path with a regular expression.  You can see an example of this code flow in the Pora application here.

If it isn’t practical to connect to an arbitrary location in your application for some reason, (e.g. the directory is hard coded everywhere, or your code is precompiled, or you are reluctant to change it) then you can also tell the broker to put the volume in a specific location when you bind the application to the service.  To do that, unbind the service, and then rebind it using the -c option.  This will create any necessary directories on the container, and put the volume mount in the specific path you need:

○ → cf unbind-service pora myvolume
Unbinding app pora from service myvolume in org pcfdev-org / space pcfdev-space as admin...
○ → cf bind-service pora myvolume -c '{"mount":"/my/specific/path"}'
Binding service myvolume to app pora in org pcfdev-org / space pcfdev-space as admin...
TIP: Use 'cf restage pora' to ensure your env variable changes take effect
○ →

Type “cf restage pora” to restart the application in this new configuration, and you should observe that the application operates as before, except that it now accesses the shared volume from “/my/specific/path”.  You can also see the new path transmitted to the application by typing “cf env pora”.

6. If You Need to Copy Data

If you need to copy existing data into your new volume in order for your application to use it, you will need to get the data onto the PCFDev host, and then copy it into the source directory that the local-volume service uses to mimic a shared volume.  In a real-world scenario, these steps wouldn’t be necessary because the volume would come from a real shared filesystem that you could mount and copy content to, but the local-volume service is a test service that mimics shared volumes for simple cloudfoundry deployments.

The key that PCFDev uses for scp/ssh is located in ~/.pcfdev/vms/key.pem.  This file must be tweaked to reduce its permissions before it can be used by scp:

○ → chmod 0600 ~/.pcfdev/vms/key.pem
○ →

Now invoke scp to copy your content across to the PCFDev VM.  The example below copies a file named “assets.tar” to the var/vcap/data directory on the VM:

○ → scp -i ~/.pcfdev/vms/key.pem ./assets.tar
assets.tar 100% 10KB 10.0KB/s 00:00
○ →

Now, use “cf dev ssh” to open a ssh session into the vm, and cd to /var/vcap/data.  You should find your file there:

○ → cf dev ssh
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.19.0-74-generic x86_64)

* Documentation:
New release '16.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

vcap@agent-id-pcfdev-0:~$ cd /var/
-bash: cd: /var/ No such file or directory
vcap@agent-id-pcfdev-0:~$ cd /var/vcap/data
vcap@agent-id-pcfdev-0:/var/vcap/data$ ls
assets.tar cloud_controller_ng executor_cache jobs nginx_cc stager tmp uaa voldrivers
bbs executor garden mysql packages sys tps upgrade_preparation_nodes volumes

“ls volumes/local/_volumes” gives us a listing of the volumes that have been created.  If you are following this tutorial, there should be only one directory in there, with a long ugly name.  Move the file into that directory, and then exit the ssh session

vcap@agent-id-pcfdev-0:/var/vcap/data$ ls volumes/local/_volumes/
vcap@agent-id-pcfdev-0:/var/vcap/data$ mv assets.tar volumes/local/_volumes/843f99cc-7638-4a0f-8209-b522928cd443/
vcap@agent-id-pcfdev-0:/var/vcap/data$ exit
○ →

Finally, invoke “cf ssh pora” to open a ssh session into the cloudfoundry app.  This will allow you to verify that the data is available to your application.  If you used the -c option to specify a mount path, you should find your content there.  If not, you will find it under “/var/vcap/data/{some-guid}”:

○ → cf ssh pora
vcap@2bts9kfetl8:~$ ls /my/specific/path/
assets.tar test.txt

7. Where to Find More Information

The best place to reach the CF Diego Persistence team is on our Slack channel here: We’re happy to hear from you with any questions you have, and to help steer you in the right direction.

If you don’t already have an account on CF slack, you can create an account here:

For an introduction to Volume Services in Cloud Foundry, refer to this document:

Or watch our various presentations this year:

CF Summit Frankfurt 2016:

Spring One Platform 2016:

CF Summit Santa Clara 2016:

Building a healthy Concourse CI pipeline for Bosh deployed products

The Cloudfoundry Diego Persistence team recently spent a fair amount of time and effort building and refactoring the CI pipeline for our Ceph filesystem, volume driver, and service broker.  The end state from this exercise, while not perfect, is nonetheless pretty darn good:  It deploys Cloudfoundry, Diego, and a Cephfs cluster, along with our volume driver and service broker.  It runs our code through unit tests, certification tests, and acceptance tests.  It keeps our deployment up to date with the latest releases of CloudFoundry and the latest development branch changes to Diego.  It does all of this with minimal rework or delay; changes in our driver/broker bosh release typically flow through the pipeline in about 10 minutes.  

But our first attempt at creating the pipeline did not work very well or very quickly, so we thought it would be worth documenting our initial assumptions, what was wrong about them, some of what we learned while fixing them.

Our First Stab at It

We started with a set of assumptions about what we could run quickly and what would run slowly, and we tried to organize our pipeline around those assumptions to make sure that the quick stuff didn’t get blocked by the slow stuff.


  • Cephfs cluster deployment is slow–it requires us to apt-get a largish list of parts and then provision a cluster.  This can take 20-30 minutes.
  • Since cluster deployment is slow, and we share a bosh release for the cephfs bosh job and our driver and broker bosh jobs, we should only trigger cephfs deployment nightly when nobody is waiting–we shouldn’t trigger it when our bosh release is updated.
  • Redeploying Cephfs is not safe–to make sure that it stays in a clean state, we should undeploy it before deploying it again.
  • CloudFoundry deployment is slow–we should not automatically pick up new CF releases because it might paralyze our pipeline during the work day.
  • The pipeline should clean up on failure–bad deployments of cephfs should get torn down automatically.


What We Eventually Learned

Our first pass at the pipeline (mostly) worked, but it was slow and inefficient.  Because we structured it to deploy some of the critical components nightly or on demand, and we tore down the ceph file system vm before redeploying it, any time we needed an update, we had to wait a long time.  In the case of cephfs, we also had to create a shadow pipeline just for manually triggering cephfs redeployment.  It turned out that most of of the assumptions above were wrong, so let’s take another look at those:


Bad Assumptions:

  • Cephfs cluster deployment is slow. This is only partially true.  Because we installed cephfs using apt-get, we were doing an end-run around Bosh package management, effectively ensuring that we would re-do work in our install script whether it was necessary or not.  We switched from apt-get to Bosh managed debian packages and that sped things up a lot.  Bosh caches packages and only fetches things that have actually changed.
  • We should only trigger cephfs deployment nightly or we will repeat slow cephfs deployments whenever code changes.  This is totally untrue.  Bosh is designed to detect changes from one version to the next, so when the broker job or the driver job changes, but cephfs hasn’t changed, deploying the cephfs job will result in a no-op for bosh.  
  • Redeploying Cephfs is not safe.  This might be partially true. In theory our ceph filesystem could get corrupted in ways that would cause the pipeline to keep failing, but treating this operation as unsafe is somewhat antithetical to cloud operations.  Bosh jobs should as much as possible be safe to redeploy without removing them.
  • CloudFoundry deployment is slow.  This is usually not true.  When there are new releases of CloudFoundry, they deploy incrementally just like other bosh deployments, so only the changed jobs will result in deployment changes.  The real culprit in slow deployment times happens when there is an update to the bosh stemcell, and bosh needs to download the stemcell before it can deploy.  In order to keep that from slowing down our pipeline during the workday, we created a “nightly stemcell” task in the pipeline that doesn’t do anything, but can only run at night.  Using the latest passed stemcell from that task, and setting the stemcell as a trigger in our deploy tasks ensures that when there is a stemcell change, our pipeline will pick it up at night, and redeploy with it, and that we will never have to wait for a stemcell download during the day:
- name: nightly
  type: time
    interval: 24h
    start: 01:00 AM -0800
    stop: 1:15 AM -0800

- name: aws-stemcell
  type: bosh-io-stemcell
    name: bosh-aws-xen-hvm-ubuntu-trusty-go_agent


- name: nightly-stemcell
  - aggregate:
    - get: nightly
      trigger: true
    - get: bosh-stemcell
      resource: aws-stemcell

- name: teardown-cephfs-cluster
  - cephfs-deploy
  - aggregate:
    - get: cephfs-bosh-release
    - get: aws-stemcell
      - nightly-stemcell
      trigger: true
    - get: deployments-runtime
  - task: teardown
    file: cephfs-bosh-release/scripts/ci/

      BOSH_TARGET: <>
      DEPLOYMENT_NAME: cephfs
  • The pipeline should clean up on failureThis is generally a bad practice.  It means that we have no way of diagnosing failures in the pipeline.  Teardown after failure also doesn’t restore the health of the pipeline, unless the deployments in question are re-deployed after, but in the case of a deployment error, that could easily result in a tight loop of deployment and undeployment, so we never did that.

Where We Ended Up

Screen Shot 2016-06-29 at 2.57.33 PM

After we corrected all of our wrong assumptions, our pipeline is in much better shape:

  • Bosh deployments are incremental and frequent.  We pick up new releases as soon as they happen, and we re-test against them, so we get early warning of failures even when we didn’t make the breaking changes.
  • Our bosh job install scripts are as much as possible idempotent.  The only undeploy jobs we have in the pipeline are manually triggered.
  • We trigger slow stemcell downloads at night when nobody is working, and stick to the same stemcells during the day to avoid slow downloads.
  • Since we share the same bosh release for 3 different deployments (broker, driver, and file system) we trigger deployment of all 3 things whenever our bosh release changes.  Since Bosh is clever about not doing anything for unchanged jobs, this is a much easier approach than trying to manage separate versions of the bosh release for different jobs.
  • We use concourse serial groups to force serialization between the tasks that deploy things and the tasks that rely on those deployments.  Serial groups are far from perfect–they operate as a simple mutex with no read/write lock semantics–but for our purposes they proved to be good enough, and they are far easier than implementing our own locks.

The yaml for our current pipeline is here for reference.


In addition to our nightly job to download stemcells, we also run a nightly task to clean up bosh releases by invoking bosh cleanup.  This is a very good idea–otherwise bosh keeps everything that’s been uploaded to it, which can quickly use up available disk space.

At some point in the future, we will probably want to add additional tasks to the pipeline to clean out our Amazon S3 buckets, but so far we haven’t done that.


A special thanks to Connor Braa who recently joined our team from the Diego team where he did a great deal of Concourse wrangling.  Connor is responsible for providing us with most of the insights in this post.

Follow Us on Twitter

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.