I’m Eric Tummers, Software Architect at Statistics Netherlands, and this is how I work


My jobs description says Software Architect. I’m guiding my team through the technical hurdles and do some programming/testing/analysing myself. We follow the scrum guide and are working towards continuous integration and deployment. We do this in small steps. Every sprint we complete means the retrospective is done in the pub.

I expect the best and I give the best. Here’s the beer. Here’s the entertainment. Now have fun. That’s an order!
RasczakLocation: Sittard, NL
Current Gig: Devops Teamlead, CBS
Word that best describes how you work: Teamwork
Current mobile device: iPhone se
Current computer: MacBook 15 inch

What apps/software/tools can’t you live without? Why?

Wunderlist: work tasks, chores, groceries, planning, every task is there. I’m aware of the acquisition by Microsoft and the planned end-of-life.
Evernote: I’m a fan for years now.
Parallels 11: Running Windows on my MacBook is a must. And of course visual studio, team foundation server, build, release manager, sql server management studio, remote desktop, powershell, and some other tools I need for work.
Alfred: keyboard shortcuts for everything. I bought the powerpack and advise you to do the same. Still on v2 though.

What’s your workspace setup like?

Work at the office is on a thin client with 24 inch screen and (wired) mouse and keyboard. The desk and chair comply with all regulations. We have a great coffee machine.

dekstop_2017

My home workspace is still my Macbook 15 inch. I’ve a new setup with logitech keyboard and mouse (MX800) and Benq 24 inch monitor (BL2420PT). Nothing fancy but the extra screen space is very welcome.

What’s your best time-saving shortcut/life hack?

Timebox. Start on a task and spent the time you’ve got to get the best result. Get someone to look at it, get feedback. Then decide if the result is final or to spent some more time.

Besides your phone and computer, what gadget can’t you live without and why?

I replaced my Magellan Echo with the Garmin FR235. It has smart watch features and a optical heart rate monitor. My phone is on mute since the Garmin notifies me of everything.
My Apple Airpods. Easy to use, small and good sound. Never leave the house without them.

What everyday thing are you better at than everyone else? What’s your secret?

Learning new things. My current project lets me implement new things (joy) Also I try to learn the things I know to my team or anyone who listens.
I have a basic understanding of how things work and try to map new things on there. For the details I have a Pluralsight subscription and black belt google skills.

What do you listen to while you work?

My alarm clock plays classical music to wake me up in the morning. The car stereo plays about everything (grunge, rock, kids stories) driving to work. When I need some focus I play drum and bass on my headphones. My ringtone is still Run riot by Camo & Krooked, although it is muted since I got the Garmin.

What are you currently reading?

The Hard Thing About Hard Things. It gives an insight into the problems a CEO has and how to overcome these. I enjoyed reading it on my last vacation and plan to read it again after I finish it.

How do you recharge? What do you do when you want to forget about work?

Spending quality time with my wife and daughters. Phone on silent, no screens, no work. Mostly piggyback riding and tea parties
Also sports like running, fitness, climbing and snowboarding to keep me fit and healthy.

Fill in the blank: I’d love to see _________ answer these same questions.

DC Rainmaker (Ray) has been on my reading list for years. His reviews about sport gadgets is amazing. If you don’t know who this is I urge you to click the link. For the rest; you know why.

What’s the best advice you’ve ever received?

someecards.com - Make a shit first draft you cannot edit a blank page
I believe this is a variant on a Hemingway quote.

Is there anything else you’d like to add that might be interesting to readers?

Learn Powershell. There is so much possible with Powershell. If you can learn one thing this year pick Powershell.

Original idea from Lifehacker.com.

Posted in Uncategorized | Tagged , , , , , , , | Leave a comment

Performance boost Neo4J ORM

We’ve created our own (very) simple ORM for saving data in Neo4J. Performance was not relevant until loading a weekly batch took about 8 days. After some tweaks we manage to load the data in just over one day. (29 hours) Here is how we tuned our code. Your milage may vary.

For those unfamiliar with Neo4J: it’s a graph database that uses cypher to manipulate data. I’ve written about it here.

Image by Gordon Johnson from Pixabay
Image by Gordon Johnson from Pixabay.com
The ORM is hosted in a aspnet webapi. In the json we receive the tree is already build and there is a single rootnode. This rootnode is saved with all related nodes. Then the relations between the rootnode and the nodes are saved. By making the calls recursive with the related nodes as rootnode the complete tree of nodes is saved this way. Again, this is a (very) simple ORM.

By logging the time needed for each recursion we found a type of node with many relations. Think thousands of relations from that one node to (leaf)nodes. That was our bottleneck. We managed to handle the node differently based on the type of the node. A special function would be called to handle the great amount of relations, but how?

Looking for the type of node in all loads we performed we noticed a relation between the number of leafnodes and the time it took to save. This relation was not linear. We put the numbers in a spreadsheet and created a plot. The optimum between number of nodes in a transaction and load-time was around 10.000. So we created a loop in the special function that would create a new transaction every 10.000 nodes.

Thanks to the logging we identified this tweak. There may be more optimisations, but this brought us back on track. Simple 20-80 rule applied :mrgreen:

Posted in Development | Tagged , , , | Leave a comment

Invoke-Webrequest UnauthorizedAccessException

Today I got an UnauthorizedAccessException when running the following script in powershell

Invoke-WebRequest -Uri $url `
     -UseDefaultCredentials `
     -UseBasicParsing ` 
     -Method Post `
     -ContentType 'application/json' `
     -InFile data.json 

There was no logging on the webserver, there was no access-denied message, there was no traffic at all. What happend?
Turns out the problem was the file (data.json) was readonly. So I added the following step to remove the readonly flag.

Set-ItemProperty data.json -name IsReadOnly -value $false

Now my post to the webservice succeeds.

Posted in Development | Tagged | Leave a comment

Load environment variables in dotnet Core 1.1 hosted in IIS

On our webserver we plan to move configuration to environment variables. We can set those environment variables in our Powershell Dsc script. But the website does not pick up the environment variables. There is an issue on github (#1864) that relates the LoadUserProfile setting of the application pool in IIS to the problem.

We deploy the website with IIS Web App Deployment Task in our release pipeline. In the Advanced tab > Additional AppCmd.exe Commands we can add the line to load the userprofile (and the environment variables)

set apppool "name_of_apppool" "-processmodel.loaduserprofile:true"
recycle apppool /apppool.name:name_of_apppool

Now the website loads the (proces/user/machine) environment variables into the configuration with the default code.

public Startup(IHostingEnvironment env)
{
  var builder = new ConfigurationBuilder()
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
        .AddEnvironmentVariables();
  Configuration = builder.Build();
}
Posted in Development | Tagged , , , | Leave a comment

Seq dashboard

We’re using seq for logging the completion of an import. To monitor the progress of importing a lot of file we’ve created a dashboard in seq.

The log entries are identified by having a property (uuid) and a date / time when the entry was logged. The query for a (completed) import looks like this:

select count(*) as count 
from stream 
where Has(uuid) 
AND @Timestamp <= DateTime('2018-12-04 07:50:00 +1') 
AND @Timestamp >= DateTime('2018-11-29 20:54 +1') limit 1

We configure the style of the query to display as value and the number of processed files is displayed. Now we set the dashboard to auto refresh every hour. This will keep the number displayed updated once every hour.

The end result is “stunning” when you realise the configuration is dead simple.

Little quirk: to filter untill a certain date/time we needed to use <=, because < only didn't work.

Posted in Development | Tagged , , | Leave a comment

Manage windows service with asp.net core webapi

We’re building a webapi for our administrators. One of the features is stopping and starting a windows service on the webserver the webapi is hosted on. We needed to allow our application pool user to manage the windows service. For this we use the sc.exe tool.

First we collected the security descriptor (SID) for the application pool user with the code below.

$account = new-object System.Security.Principal.NTAccount("apppooluser");
$account.Translate([System.Security.Principal.SecurityIdentifier]).Value;
# output S-1-5-21-1003698448-2199609630-2888039566-500

Now we used se.exe to get and set the ACE for the windows service. The output of the first statement is used in the second statement, with the SID from above added before the S:

sc.exe sdshow Spooler
# output "D:(A;;CCLCSWLOCRRC;;;AU)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)"
# add the apppooluser to the list in the D: section
sc.exe sdset Spooler "D:(A;;CCLCSWLOCRRC;;;AU)(A;;RPWPCR;;;S-1-5-21-1003698448-2199609630-2888039566-500)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)"
# [SC] SetServiceObjectSecurity SUCCESS

Breakdown of the setting:

Code Explained
D: dacl_flags are used
A ACCESS_ALLOWED_ACE_TYPE
RPWPCR ADS_RIGHT_DS_READ_PROP ADS_RIGHT_DS_WRITE_PROP ADS_RIGHT_DS_CONTROL_ACCESS
S-1-5-21-1003698448-2199609630-2888039566-500 SID of the apppooluser

Now we can start and stop the windows service from our webapi. We added authentication and authorisation to the webapi to control who can perform the actions. Check-in and release this features. Done.

References

Security Descriptior Definition Language
ACE strings
Sc sdset
SC.exe

Posted in Development, Security | Tagged , , , , | Leave a comment

Seq

Seq creates the visibility you need to quickly identify and diagnose problems in complex applications and microservices. Empower your team to build better software by centralizing, searching, and alerting on structured application logs.
https://getseq.net

source https://getseq.net
Image source https://getseq.net

Every project we used to create our own logging tables and GUI to access those tables. No More!

We now install Seq in the first sprint and start logging. Seq provides the right toolset to investigate issues, gives insights we never seen before and is free if you can live with the limitations. Purging the logging after a set timeinterval is part of the features. And everything is done from a web interface.

So far we’ve learned that you need to log on different levels and you’ll need a structured logging nuget like serilog.

Want to start logging too? Here are some links:

Posted in Development, Tooling | Tagged , , | Leave a comment

Serilog


Image courtesy of Ricardo Gomez Angel / unsplash.com

https://www.nuget.org/packages/serilog

With Serilog we can log with named properties. The advantage is that processing of the log(file) is no longer needed as it is already structured. You can stil output plain text though.

Posted in Development, Tooling | Tagged , , | Leave a comment