Performance boost Neo4J ORM

We’ve created our own (very) simple ORM for saving data in Neo4J. Performance was not relevant until loading a weekly batch took about 8 days. After some tweaks we manage to load the data in just over one day. (29 hours) Here is how we tuned our code. Your milage may vary.

For those unfamiliar with Neo4J: it’s a graph database that uses cypher to manipulate data. I’ve written about it here.

Image by Gordon Johnson from Pixabay
Image by Gordon Johnson from Pixabay.com
The ORM is hosted in a aspnet webapi. In the json we receive the tree is already build and there is a single rootnode. This rootnode is saved with all related nodes. Then the relations between the rootnode and the nodes are saved. By making the calls recursive with the related nodes as rootnode the complete tree of nodes is saved this way. Again, this is a (very) simple ORM.

By logging the time needed for each recursion we found a type of node with many relations. Think thousands of relations from that one node to (leaf)nodes. That was our bottleneck. We managed to handle the node differently based on the type of the node. A special function would be called to handle the great amount of relations, but how?

Looking for the type of node in all loads we performed we noticed a relation between the number of leafnodes and the time it took to save. This relation was not linear. We put the numbers in a spreadsheet and created a plot. The optimum between number of nodes in a transaction and load-time was around 10.000. So we created a loop in the special function that would create a new transaction every 10.000 nodes.

Thanks to the logging we identified this tweak. There may be more optimisations, but this brought us back on track. Simple 20-80 rule applied :mrgreen:

Posted in Development | Tagged , , , | Leave a comment

Invoke-Webrequest UnauthorizedAccessException

Today I got an UnauthorizedAccessException when running the following script in powershell

Invoke-WebRequest -Uri $url `
     -UseDefaultCredentials `
     -UseBasicParsing ` 
     -Method Post `
     -ContentType 'application/json' `
     -InFile data.json 

There was no logging on the webserver, there was no access-denied message, there was no traffic at all. What happend?
Turns out the problem was the file (data.json) was readonly. So I added the following step to remove the readonly flag.

Set-ItemProperty data.json -name IsReadOnly -value $false

Now my post to the webservice succeeds.

Posted in Development | Tagged | Leave a comment

Load environment variables in dotnet Core 1.1 hosted in IIS

On our webserver we plan to move configuration to environment variables. We can set those environment variables in our Powershell Dsc script. But the website does not pick up the environment variables. There is an issue on github (#1864) that relates the LoadUserProfile setting of the application pool in IIS to the problem.

We deploy the website with IIS Web App Deployment Task in our release pipeline. In the Advanced tab > Additional AppCmd.exe Commands we can add the line to load the userprofile (and the environment variables)

set apppool "name_of_apppool" "-processmodel.loaduserprofile:true"
recycle apppool /apppool.name:name_of_apppool

Now the website loads the (proces/user/machine) environment variables into the configuration with the default code.

public Startup(IHostingEnvironment env)
{
  var builder = new ConfigurationBuilder()
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
        .AddEnvironmentVariables();
  Configuration = builder.Build();
}
Posted in Development | Tagged , , , | Leave a comment

Seq dashboard

We’re using seq for logging the completion of an import. To monitor the progress of importing a lot of file we’ve created a dashboard in seq.

The log entries are identified by having a property (uuid) and a date / time when the entry was logged. The query for a (completed) import looks like this:

select count(*) as count 
from stream 
where Has(uuid) 
AND @Timestamp <= DateTime('2018-12-04 07:50:00 +1') 
AND @Timestamp >= DateTime('2018-11-29 20:54 +1') limit 1

We configure the style of the query to display as value and the number of processed files is displayed. Now we set the dashboard to auto refresh every hour. This will keep the number displayed updated once every hour.

The end result is “stunning” when you realise the configuration is dead simple.

Little quirk: to filter untill a certain date/time we needed to use <=, because < only didn't work.

Posted in Development | Tagged , , | Leave a comment

Manage windows service with asp.net core webapi

We’re building a webapi for our administrators. One of the features is stopping and starting a windows service on the webserver the webapi is hosted on. We needed to allow our application pool user to manage the windows service. For this we use the sc.exe tool.

First we collected the security descriptor (SID) for the application pool user with the code below.

$account = new-object System.Security.Principal.NTAccount("apppooluser");
$account.Translate([System.Security.Principal.SecurityIdentifier]).Value;
# output S-1-5-21-1003698448-2199609630-2888039566-500

Now we used se.exe to get and set the ACE for the windows service. The output of the first statement is used in the second statement, with the SID from above added before the S:

sc.exe sdshow Spooler
# output "D:(A;;CCLCSWLOCRRC;;;AU)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)"
# add the apppooluser to the list in the D: section
sc.exe sdset Spooler "D:(A;;CCLCSWLOCRRC;;;AU)(A;;RPWPCR;;;S-1-5-21-1003698448-2199609630-2888039566-500)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)"
# [SC] SetServiceObjectSecurity SUCCESS

Breakdown of the setting:

Code Explained
D: dacl_flags are used
A ACCESS_ALLOWED_ACE_TYPE
RPWPCR ADS_RIGHT_DS_READ_PROP ADS_RIGHT_DS_WRITE_PROP ADS_RIGHT_DS_CONTROL_ACCESS
S-1-5-21-1003698448-2199609630-2888039566-500 SID of the apppooluser

Now we can start and stop the windows service from our webapi. We added authentication and authorisation to the webapi to control who can perform the actions. Check-in and release this features. Done.

References

Security Descriptior Definition Language
ACE strings
Sc sdset
SC.exe

Posted in Development, Security | Tagged , , , , | Leave a comment

Seq

Seq creates the visibility you need to quickly identify and diagnose problems in complex applications and microservices. Empower your team to build better software by centralizing, searching, and alerting on structured application logs.
https://getseq.net

source https://getseq.net
Image source https://getseq.net

Every project we used to create our own logging tables and GUI to access those tables. No More!

We now install Seq in the first sprint and start logging. Seq provides the right toolset to investigate issues, gives insights we never seen before and is free if you can live with the limitations. Purging the logging after a set timeinterval is part of the features. And everything is done from a web interface.

So far we’ve learned that you need to log on different levels and you’ll need a structured logging nuget like serilog.

Want to start logging too? Here are some links:

Posted in Development, Tooling | Tagged , , | Leave a comment

Serilog


Image courtesy of Ricardo Gomez Angel / unsplash.com

https://www.nuget.org/packages/serilog

With Serilog we can log with named properties. The advantage is that processing of the log(file) is no longer needed as it is already structured. You can stil output plain text though.

Posted in Development, Tooling | Tagged , , | Leave a comment