Deploy your first dotnet core app to Cloud Foundry

After installing Cloud Foundry it is time to deploy your first dotnet core app.

Make sure you’re using the right framework/sdk version. If the version is not supported by Cloud Foundry you’ll get logging like below

-----> Dotnet-Core Buildpack version 2.0.1
-----> Supplying Dotnet Core
-----> Installing libunwind 1.2.1
       Download [https://buildpacks.cloudfoundry.org/dependencies/manual-binaries/dotnet/libunwind-1.2.1-linux-x64-80af276a.tgz]
-----> Installing dotnet 2.0.3
       Download [https://buildpacks.cloudfoundry.org/dependencies/dotnet/dotnet.2.0.3.linux-amd64-b56d13fc.tar.xz]
-----> Finalizing Dotnet Core
-----> Restore dotnet dependencies
       /tmp/contents738113418/deps/0/dotnet/sdk/2.0.3/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.TargetFrameworkInference.targets(135,5): 
error : 
The current .NET SDK does not support targeting .NET Core 2.2.  
Either target .NET Core 2.0 or lower, or use a version of the 
.NET SDK that supports .NET Core 2.2. [/tmp/app/newcore.csproj]
       /tmp/contents738113418/deps/0/dotnet/sdk/2.0.3/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.TargetFrameworkInference.targets(135,5): error : The current .NET SDK does not support targeting .NET Core 2.2.  Either target .NET Core 2.0 or lower, or use a version of the .NET SDK that supports .NET Core 2.2. [/tmp/app/newcore.csproj]
       **ERROR** Unable to run dotnet restore: exit status 1
Failed to compile droplet: Failed to run finalize script: exit status 12
Exit status 223
Stopping instance 872072e8-a3db-4b10-94ba-9dd207587dcc
Destroying container
Error staging application: App staging failed in the buildpack compile phase
FAILED

You can force the use of a supported dotnet core version by adding a global.json to the root of your project. This can be generated by the dotnet cli.

{
   "sdk": {
      "version": "2.0.3"
   }
}

Now create a simple website with the cli and push it to cloud foundry. You should be greeted with success messages 😉

mkdir core
cd core
dotnet new global --sdk-version 2.0.3
dotnet new mvc
cf push core

In Stratos the new app is visible. There you can view logging, scale up/down and configure routes.

The new app is available on http://core.bosh-lite.com (default route)

Posted in Development | Tagged , | Leave a comment

Install Cloud Foundry on macOS with Virtualbox

We’re investigating Cloud Foundry for our PAAS solution. At home I’ve installed it on my mac with virtualbox. This post is a combination of information available elsewhere (links are included) and can be used to get you’re own local Cloud Foundry up and running.

Prerequisites

Install VirtualBox (https://www.virtualbox.org/wiki/Download_Old_Builds_5_2)
Install VirtualBox extension pack
Install xcode

xcode-select --install

Install Homebrew (https://brew.sh) and use it to install bosh-cli, openssl and node.js

brew install cloudfoundry/tap/bosh-cli
brew install openssl
brew install node

Now start installation …

Bosh-lite

Follow https://bosh.io/docs/bosh-lite/ and you’ll have bosh-lite running. JUST DON’T INSTALL ZOOKEEPER! The script I’ve used is below. The last line adds a route to the system that is crucial for the cloud foundry cli to work.

git clone https://github.com/cloudfoundry/bosh-deployment ~/workspace/bosh-deployment
mkdir -p ~/deployments/vbox
cd ~/deployments/vbox
bosh create-env ~/workspace/bosh-deployment/bosh.yml \
  --state ./state.json \
  -o ~/workspace/bosh-deployment/virtualbox/cpi.yml \
  -o ~/workspace/bosh-deployment/virtualbox/outbound-network.yml \
  -o ~/workspace/bosh-deployment/bosh-lite.yml \
  -o ~/workspace/bosh-deployment/bosh-lite-runc.yml \
  -o ~/workspace/bosh-deployment/uaa.yml \
  -o ~/workspace/bosh-deployment/credhub.yml \
  -o ~/workspace/bosh-deployment/jumpbox-user.yml \
  --vars-store ./creds.yml \
  -v director_name=bosh-lite \
  -v internal_ip=192.168.50.6 \
  -v internal_gw=192.168.50.1 \
  -v internal_cidr=192.168.50.0/24 \
  -v outbound_network_name=NatNetwork
bosh alias-env vbox -e 192.168.50.6 --ca-cert <(bosh int ./creds.yml --path /director_ssl/ca)
export BOSH_CLIENT=admin
export BOSH_CLIENT_SECRET=`bosh int ./creds.yml --path /admin_password`
sudo route add -net 10.244.0.0/16     192.168.50.6 # Mac OS X

Congratulations you’ve now got a working bosh-lite installation. You can use the bosh-cli to interact with it. Now install/deploy Cloud Foundry.

Cloud Foundry

Read up on http://operator-workshop.cloudfoundry.org/labs/deploy-cf/ but use the script below as it fills some holes.

git clone https://github.com/cloudfoundry/cf-deployment ~/workspace/cf-deployment
cd ~/workspace/cf-deployment
git checkout tags/v1.15.0
export BOSH_ENVIRONMENT=192.168.50.6
bosh update-cloud-config ~/workspace/cf-deployment/iaas-support/bosh-lite/cloud-config.yml
export STEMCELL_VERSION=$(bosh int cf-deployment.yml --path '/stemcells/alias=default/version')
bosh upload-stemcell "https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent?v=$STEMCELL_VERSION"
export SYSTEM_DOMAIN=bosh-lite.com
bosh -d cf deploy ~/workspace/cf-deployment/cf-deployment.yml \
-o ~/workspace/cf-deployment/operations/bosh-lite.yml \
-o ~/workspace/cf-deployment/operations/use-compiled-releases.yml \
--vars-store deployment-vars.yml \
-v system_domain=$SYSTEM_DOMAIN

Now login to you’re Cloud Foundry environment. The password for the admin user is revealed with the first statement of the script below.
More info on (http://operator-workshop.cloudfoundry.org/labs/cf-basics/):

bosh interpolate --path /cf_admin_password ~/workspace/cf-deployment/deployment-vars.yml
cf login -a https://api.$SYSTEM_DOMAIN --skip-ssl-validation
cf create-org demo
cf target -o demo
cf create-space dev
cf target -o demo -s dev

Stratos

A nice GUI for your Cloud Foundry is Stratos. You deploy Stratos to CF with details from https://github.com/cloudfoundry-incubator/stratos/tree/v2-master/deploy/cloud-foundry This is where node.js is needed.

cd ~/workspace
git clone https://github.com/cloudfoundry-incubator/stratos
cd stratos
git checkout tags/stable -b stable
npm install
npm run prebuild-ui
cf push

Now logon to Stratos on https://console.bosh-lite.com with Admin and the password from the first line in the previous script.

Save state the VirtualBox machine before shutdown to preserve your environment. After a restart create the route again.

Extra

If you’ve already installed node, make sure to update it to the latest version:

brew update
brew upgrade node
npm update -g npm
Posted in Development | Tagged , , , , | 1 Comment

Performance boost Neo4J ORM

We’ve created our own (very) simple ORM for saving data in Neo4J. Performance was not relevant until loading a weekly batch took about 8 days. After some tweaks we manage to load the data in just over one day. (29 hours) Here is how we tuned our code. Your milage may vary.

For those unfamiliar with Neo4J: it’s a graph database that uses cypher to manipulate data. I’ve written about it here.

Image by Gordon Johnson from Pixabay
Image by Gordon Johnson from Pixabay.com
The ORM is hosted in a aspnet webapi. In the json we receive the tree is already build and there is a single rootnode. This rootnode is saved with all related nodes. Then the relations between the rootnode and the nodes are saved. By making the calls recursive with the related nodes as rootnode the complete tree of nodes is saved this way. Again, this is a (very) simple ORM.

By logging the time needed for each recursion we found a type of node with many relations. Think thousands of relations from that one node to (leaf)nodes. That was our bottleneck. We managed to handle the node differently based on the type of the node. A special function would be called to handle the great amount of relations, but how?

Looking for the type of node in all loads we performed we noticed a relation between the number of leafnodes and the time it took to save. This relation was not linear. We put the numbers in a spreadsheet and created a plot. The optimum between number of nodes in a transaction and load-time was around 10.000. So we created a loop in the special function that would create a new transaction every 10.000 nodes.

Thanks to the logging we identified this tweak. There may be more optimisations, but this brought us back on track. Simple 20-80 rule applied :mrgreen:

Posted in Development | Tagged , , , | Leave a comment

Invoke-Webrequest UnauthorizedAccessException

Today I got an UnauthorizedAccessException when running the following script in powershell

Invoke-WebRequest -Uri $url `
     -UseDefaultCredentials `
     -UseBasicParsing ` 
     -Method Post `
     -ContentType 'application/json' `
     -InFile data.json 

There was no logging on the webserver, there was no access-denied message, there was no traffic at all. What happend?
Turns out the problem was the file (data.json) was readonly. So I added the following step to remove the readonly flag.

Set-ItemProperty data.json -name IsReadOnly -value $false

Now my post to the webservice succeeds.

Posted in Development | Tagged | Leave a comment

Load environment variables in dotnet Core 1.1 hosted in IIS

On our webserver we plan to move configuration to environment variables. We can set those environment variables in our Powershell Dsc script. But the website does not pick up the environment variables. There is an issue on github (#1864) that relates the LoadUserProfile setting of the application pool in IIS to the problem.

We deploy the website with IIS Web App Deployment Task in our release pipeline. In the Advanced tab > Additional AppCmd.exe Commands we can add the line to load the userprofile (and the environment variables)

set apppool "name_of_apppool" "-processmodel.loaduserprofile:true"
recycle apppool /apppool.name:name_of_apppool

Now the website loads the (proces/user/machine) environment variables into the configuration with the default code.

public Startup(IHostingEnvironment env)
{
  var builder = new ConfigurationBuilder()
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
        .AddEnvironmentVariables();
  Configuration = builder.Build();
}
Posted in Development | Tagged , , , | Leave a comment

Seq dashboard

We’re using seq for logging the completion of an import. To monitor the progress of importing a lot of file we’ve created a dashboard in seq.

The log entries are identified by having a property (uuid) and a date / time when the entry was logged. The query for a (completed) import looks like this:

select count(*) as count 
from stream 
where Has(uuid) 
AND @Timestamp <= DateTime('2018-12-04 07:50:00 +1') 
AND @Timestamp >= DateTime('2018-11-29 20:54 +1') limit 1

We configure the style of the query to display as value and the number of processed files is displayed. Now we set the dashboard to auto refresh every hour. This will keep the number displayed updated once every hour.

The end result is “stunning” when you realise the configuration is dead simple.

Little quirk: to filter untill a certain date/time we needed to use <=, because < only didn't work.

Posted in Development | Tagged , , | Leave a comment

Manage windows service with asp.net core webapi

We’re building a webapi for our administrators. One of the features is stopping and starting a windows service on the webserver the webapi is hosted on. We needed to allow our application pool user to manage the windows service. For this we use the sc.exe tool.

First we collected the security descriptor (SID) for the application pool user with the code below.

$account = new-object System.Security.Principal.NTAccount("apppooluser");
$account.Translate([System.Security.Principal.SecurityIdentifier]).Value;
# output S-1-5-21-1003698448-2199609630-2888039566-500

Now we used se.exe to get and set the ACE for the windows service. The output of the first statement is used in the second statement, with the SID from above added before the S:

sc.exe sdshow Spooler
# output "D:(A;;CCLCSWLOCRRC;;;AU)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)"
# add the apppooluser to the list in the D: section
sc.exe sdset Spooler "D:(A;;CCLCSWLOCRRC;;;AU)(A;;RPWPCR;;;S-1-5-21-1003698448-2199609630-2888039566-500)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)"
# [SC] SetServiceObjectSecurity SUCCESS

Breakdown of the setting:

Code Explained
D: dacl_flags are used
A ACCESS_ALLOWED_ACE_TYPE
RPWPCR ADS_RIGHT_DS_READ_PROP ADS_RIGHT_DS_WRITE_PROP ADS_RIGHT_DS_CONTROL_ACCESS
S-1-5-21-1003698448-2199609630-2888039566-500 SID of the apppooluser

Now we can start and stop the windows service from our webapi. We added authentication and authorisation to the webapi to control who can perform the actions. Check-in and release this features. Done.

References

Security Descriptior Definition Language
ACE strings
Sc sdset
SC.exe

Posted in Development, Security | Tagged , , , , | Leave a comment