Offline buildpack for Cloud Foundry

Posted on April 26, 2019 by erictummers

This post is a combination of information available elsewhere (links are included) and can be used to get you’re own offline buildpack on Cloud Foundry.

Prerequisites

install go (https://ahmadawais.com/install-go-lang-on-macos-with-homebrew/)

Dotnet-core buildpack

Instructions on https://github.com/cloudfoundry/dotnet-core-buildpack/tree/v2.0.1

cd ~/workspace
git clone https://github.com/cloudfoundry/dotnet-core-buildpack
cd dotnet-core-buildpack
git checkout v2.0.1 #I'm on v2.0.1, you may need another version
export GOPATH="${HOME}/.go"
export GOROOT="$(brew --prefix golang)/libexec"
export PATH="$PATH:${GOPATH}/bin:${GOROOT}/bin"
source .envrc
(cd src/*/vendor/github.com/cloudfoundry/libbuildpack/packager/buildpack-packager && go install)
buildpack-packager --cached
cf create-buildpack offline-dotnet-core dotnet-core_buildpack-cached-v2.0.1.zip 1

Warning! buildpack is 841MB.

Now you can use you’re offline buildpack with the -b parameter:

cf push core -b offline-dotnet-core
Posted in Development | Tagged , | Leave a comment

Deploy your first dotnet core app to Cloud Foundry

Posted on April 24, 2019 by erictummers

After installing Cloud Foundry it is time to deploy your first dotnet core app.

Make sure you’re using the right framework/sdk version. If the version is not supported by Cloud Foundry you’ll get logging like below

-----> Dotnet-Core Buildpack version 2.0.1
-----> Supplying Dotnet Core
-----> Installing libunwind 1.2.1
       Download [https://buildpacks.cloudfoundry.org/dependencies/manual-binaries/dotnet/libunwind-1.2.1-linux-x64-80af276a.tgz]
-----> Installing dotnet 2.0.3
       Download [https://buildpacks.cloudfoundry.org/dependencies/dotnet/dotnet.2.0.3.linux-amd64-b56d13fc.tar.xz]
-----> Finalizing Dotnet Core
-----> Restore dotnet dependencies
       /tmp/contents738113418/deps/0/dotnet/sdk/2.0.3/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.TargetFrameworkInference.targets(135,5): 
error : 
The current .NET SDK does not support targeting .NET Core 2.2.  
Either target .NET Core 2.0 or lower, or use a version of the 
.NET SDK that supports .NET Core 2.2. [/tmp/app/newcore.csproj]
       /tmp/contents738113418/deps/0/dotnet/sdk/2.0.3/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.TargetFrameworkInference.targets(135,5): error : The current .NET SDK does not support targeting .NET Core 2.2.  Either target .NET Core 2.0 or lower, or use a version of the .NET SDK that supports .NET Core 2.2. [/tmp/app/newcore.csproj]
       **ERROR** Unable to run dotnet restore: exit status 1
Failed to compile droplet: Failed to run finalize script: exit status 12
Exit status 223
Stopping instance 872072e8-a3db-4b10-94ba-9dd207587dcc
Destroying container
Error staging application: App staging failed in the buildpack compile phase
FAILED

You can force the use of a supported dotnet core version by adding a global.json to the root of your project. This can be generated by the dotnet cli.

{
   "sdk": {
      "version": "2.0.3"
   }
}

Now create a simple website with the cli and push it to cloud foundry. You should be greeted with success messages 😉

mkdir core
cd core
dotnet new global --sdk-version 2.0.3
dotnet new mvc
cf push core

In Stratos the new app is visible. There you can view logging, scale up/down and configure routes.

The new app is available on http://core.bosh-lite.com (default route)

Posted in Development | Tagged , | Leave a comment

Install Cloud Foundry on macOS with Virtualbox

Posted on April 17, 2019 by erictummers

We’re investigating Cloud Foundry for our PAAS solution. At home I’ve installed it on my mac with virtualbox. This post is a combination of information available elsewhere (links are included) and can be used to get you’re own local Cloud Foundry up and running.

Prerequisites

Install VirtualBox (https://www.virtualbox.org/wiki/Download_Old_Builds_5_2)
Install VirtualBox extension pack
Install xcode

xcode-select --install

Install Homebrew (https://brew.sh) and use it to install bosh-cli, openssl and node.js

brew install cloudfoundry/tap/bosh-cli
brew install openssl
brew install node

Now start installation …

Bosh-lite

Follow https://bosh.io/docs/bosh-lite/ and you’ll have bosh-lite running. JUST DON’T INSTALL ZOOKEEPER! The script I’ve used is below. The last line adds a route to the system that is crucial for the cloud foundry cli to work.

git clone https://github.com/cloudfoundry/bosh-deployment ~/workspace/bosh-deployment
mkdir -p ~/deployments/vbox
cd ~/deployments/vbox
bosh create-env ~/workspace/bosh-deployment/bosh.yml \
  --state ./state.json \
  -o ~/workspace/bosh-deployment/virtualbox/cpi.yml \
  -o ~/workspace/bosh-deployment/virtualbox/outbound-network.yml \
  -o ~/workspace/bosh-deployment/bosh-lite.yml \
  -o ~/workspace/bosh-deployment/bosh-lite-runc.yml \
  -o ~/workspace/bosh-deployment/uaa.yml \
  -o ~/workspace/bosh-deployment/credhub.yml \
  -o ~/workspace/bosh-deployment/jumpbox-user.yml \
  --vars-store ./creds.yml \
  -v director_name=bosh-lite \
  -v internal_ip=192.168.50.6 \
  -v internal_gw=192.168.50.1 \
  -v internal_cidr=192.168.50.0/24 \
  -v outbound_network_name=NatNetwork
bosh alias-env vbox -e 192.168.50.6 --ca-cert <(bosh int ./creds.yml --path /director_ssl/ca)
export BOSH_CLIENT=admin
export BOSH_CLIENT_SECRET=`bosh int ./creds.yml --path /admin_password`
sudo route add -net 10.244.0.0/16     192.168.50.6 # Mac OS X

Congratulations you’ve now got a working bosh-lite installation. You can use the bosh-cli to interact with it. Now install/deploy Cloud Foundry.

Cloud Foundry

Read up on http://operator-workshop.cloudfoundry.org/labs/deploy-cf/ but use the script below as it fills some holes.

git clone https://github.com/cloudfoundry/cf-deployment ~/workspace/cf-deployment
cd ~/workspace/cf-deployment
git checkout tags/v1.15.0
export BOSH_ENVIRONMENT=192.168.50.6
bosh update-cloud-config ~/workspace/cf-deployment/iaas-support/bosh-lite/cloud-config.yml
export STEMCELL_VERSION=$(bosh int cf-deployment.yml --path '/stemcells/alias=default/version')
bosh upload-stemcell "https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent?v=$STEMCELL_VERSION"
export SYSTEM_DOMAIN=bosh-lite.com
bosh -d cf deploy ~/workspace/cf-deployment/cf-deployment.yml \
-o ~/workspace/cf-deployment/operations/bosh-lite.yml \
-o ~/workspace/cf-deployment/operations/use-compiled-releases.yml \
--vars-store deployment-vars.yml \
-v system_domain=$SYSTEM_DOMAIN

Now login to you’re Cloud Foundry environment. The password for the admin user is revealed with the first statement of the script below.
More info on (http://operator-workshop.cloudfoundry.org/labs/cf-basics/):

bosh interpolate --path /cf_admin_password ~/workspace/cf-deployment/deployment-vars.yml
cf login -a https://api.$SYSTEM_DOMAIN --skip-ssl-validation
cf create-org demo
cf target -o demo
cf create-space dev
cf target -o demo -s dev

Stratos

A nice GUI for your Cloud Foundry is Stratos. You deploy Stratos to CF with details from https://github.com/cloudfoundry-incubator/stratos/tree/v2-master/deploy/cloud-foundry This is where node.js is needed.

cd ~/workspace
git clone https://github.com/cloudfoundry-incubator/stratos
cd stratos
git checkout tags/stable -b stable
npm install
npm run prebuild-ui
cf push

Now logon to Stratos on https://console.bosh-lite.com with Admin and the password from the first line in the previous script.

Save state the VirtualBox machine before shutdown to preserve your environment. After a restart create the route again.

Extra

If you’ve already installed node, make sure to update it to the latest version:

brew update
brew upgrade node
npm update -g npm
Posted in Development | Tagged , , , , | 7 Comments

Performance boost Neo4J ORM

Posted on March 20, 2019 by erictummers

We’ve created our own (very) simple ORM for saving data in Neo4J. Performance was not relevant until loading a weekly batch took about 8 days. After some tweaks we manage to load the data in just over one day. (29 hours) Here is how we tuned our code. Your milage may vary.

For those unfamiliar with Neo4J: it’s a graph database that uses cypher to manipulate data. I’ve written about it here.

Image by Gordon Johnson from Pixabay
Image by Gordon Johnson from Pixabay.com
The ORM is hosted in a aspnet webapi. In the json we receive the tree is already build and there is a single rootnode. This rootnode is saved with all related nodes. Then the relations between the rootnode and the nodes are saved. By making the calls recursive with the related nodes as rootnode the complete tree of nodes is saved this way. Again, this is a (very) simple ORM.

By logging the time needed for each recursion we found a type of node with many relations. Think thousands of relations from that one node to (leaf)nodes. That was our bottleneck. We managed to handle the node differently based on the type of the node. A special function would be called to handle the great amount of relations, but how?

Looking for the type of node in all loads we performed we noticed a relation between the number of leafnodes and the time it took to save. This relation was not linear. We put the numbers in a spreadsheet and created a plot. The optimum between number of nodes in a transaction and load-time was around 10.000. So we created a loop in the special function that would create a new transaction every 10.000 nodes.

Thanks to the logging we identified this tweak. There may be more optimisations, but this brought us back on track. Simple 20-80 rule applied :mrgreen:

Posted in Development | Tagged , , , | Leave a comment

Invoke-Webrequest UnauthorizedAccessException

Posted on February 20, 2019 by erictummers

Today I got an UnauthorizedAccessException when running the following script in powershell

Invoke-WebRequest -Uri $url `
     -UseDefaultCredentials `
     -UseBasicParsing ` 
     -Method Post `
     -ContentType 'application/json' `
     -InFile data.json

There was no logging on the webserver, there was no access-denied message, there was no traffic at all. What happend?
Turns out the problem was the file (data.json) was readonly. So I added the following step to remove the readonly flag.

Set-ItemProperty data.json -name IsReadOnly -value $false

Now my post to the webservice succeeds.

Posted in Development | Tagged | Leave a comment