Tuesday, 4 October 2016

Improve Code Quality with NDepend

One of the most important things you could do to increase the code quality of your product and bring your team up to speed in terms of code reviews is to leverage the code quality analysis to a third party component. Nowadays I rely on FXCop (a free static code analysis tool from Microsoft that checks .NET managed code assemblies for conformance to Microsoft's .NET Framework Design Guidelines. source: wikipedia) to catch any code smell for me but this time I'm bringing an additional player into the game: NDepend.

NDepend "is a static analysis tool for .NET managed code. This tool supports a large number of code metrics, allows for visualization of dependencies using directed graphs and dependency matrix. The tools also performs code base snapshots comparison, and validation of architectural and quality rules. User-defined rules can be written using LINQ queries. This possibility is named CQLinq. The tool also comes with a large number of predefined CQLinq code rules. Code rules can be checked automatically in Visual Studio or during continuous integration". source: wikipedia.

Review your code!
The best time to find any potential issues in code is as soon as the code is written. "If you let code sit and rot for a while, it won't smell any prettier" by S. Venkat, H Andy on Practices of an Agile Developer.
"Formal code inspections are about as twice as efficient as any known form of testing in finding deep and obscure programming bugs and are the only known methid to top 80% in defect-removal efficiency." by Capers Jones in Estimating Software Costs.
As pointed out, code reviews are the best way to find and solve problems. Make sure you include a code analysis step during your continuous integration/continuous delivery model so issues are spotted earlier rather than later. Below you can see a simple pipeline that I tend to follow for any of my builds. This pipeline only refers to the commit stage and following the best practices to provide fast and provide useful feedback.

When any code is checked-in to git, tfs, subversion, etc your build agent aka TeamCity, Cruise Control, Jenkins, etc. should pick up those changes and start processing that code through the pipeline. During this process, notifications should be sent to the user when something "breaks". In my case, anytime additional code duplicates or code smells increase during the build, the pipeline stops building and reports back to the user with the specific problem. I believe this is crucial to spot any possible issues in the code and that will help the developer to take ownership of the code.

During the Code Analysis step, I rely on FXCop and NDepend to provide a suite of principles and practices that will make the code follow certain standards that are adopted by thousands of other developers. FXCop only provides a fair list of code inspections which are really useful but NDepend can give you more.  Let's see what NDepend can do for you! (note that I'm assuming that you know FXCop quite well).

Integrating NDepend with Visual Studio
NDepend extension is really easy to install and integrate with your VS. Download the latest from here and follow the steps in this video for its integration.

Integrating NDepend with TeamCity
This is the bit that interests me the most. Thanks to the easy integration of NDepend with TeamCity I can leverage the code analysis step to focus on the final code review as the code should reach a common quality standard. To integrate NDepend with TeamCity, you can follow the steps in the following tutorial.

Once configured you should see NDepend under your build runners:
The next step is related to the project configuration itself. NDepend build step expects an NDepend Project file. To create one, you will have to do it in Visual Studio and check it in as part of the solution to your repository. Then TeamCity should see it as part of the source code.

Example of NDepend project as part of the source code:

To create a new project, go to VS and NDepend -> Project -> New Project and create a project under your main solution. Then you'll have to configure the dependencies that you want to attach to the project and once that's done you'll be able to explore the nice report that gets generated out of it.

Example of configuration:

Notice that I had to open the project file to change the paths to relative paths as TeamCity works via relative paths and I was getting an error when running the project file.

Once you have your project configured, run Ndepend from your VS and generate your report:

From this html Report, you can get all the summary metrics about your project and by clicking on them you can drill into the items to expand on those results.

Once you configure NDepend in your TeamCity build step, you will get the same information with the amount of code inspections and errors that are encountered during the build:
The number of inspections appear on the build results so TeamCity can keep track of it overtime:

Once you drill into the build results, you will see the code inspections results (with all the findings from NDepend):
And also the same report you saw in VS:

Now that our system is up and running and the integration of NDepend is done, we can focus on the important stuff...code quality.

One of the things I like the most is the way code rules are set up. It's so easy to enable/disable rules that it's all done in the NDepend Project file. So if you are not agreeing with one particular rule for code quality, just disable it and it won't bother you anymore (try to disable one of the rules for FXCop...and you tell me how you did it and how long it took you..):

Here the full list of code metric definitions.

The main features of NDepend are listed below:
  • Dependency Graph
  • Dependency Matrix
  • Treemap Metric View
  • Abstractness vs. Instability
  • Code Query
Dependency Graph
This very useful diagram allows you to see the relationship between different objects in your solution/s.

Dependency Matrix
The Dependency matrix gives you a coupling score matrix. This will help you identify highly coupled dependencies and if you click on the number, you can see the number of relationships for that particular entry:

Treemap Metric View
In this spectacular view, your source code is mapped into nested rectangles showing methods that are relative to other methods. This will give you a great list of candidates for refactoring. In my case I have few boxes with a cyclomatic complexity of 6 units which tell me that are good candidates for my review and analysis. Once you click on one of this rectangles the source code is shown.

Abstractness vs. Instability
This graph give us a high level overview of our application in terms of where it is heading (too abstract or too unstable).

Here is how to read this chart:

  • If an assembly is very stable (that is, lots of assemblies depend on it) and it's not extensible (no abstract classes, no virtuals, etc) then you're in the lower-left quadrant of the chart in the well-named Zone of Pain.
  • If an assembly is very abstract, very extensible, but no one depends on it (it's not really being used) then it moves towards the Zone of Uselessness.

Code Query
Code Query is NDepend tool that uses its scripting language (CQLinq). This allows users to write their own queries and rules. I like the fact that it is highly customisable and developers love scripting components. I haven't played much with it but I can see the potential for it.

I do believe that NDepend needs to be included in your tool-belt kit. Once you start using it you can't leave without it. Just the amount of information that handles for you is amazing and it gives you that level of confidence that all your developers are writing code with the same standards and code practices out there. It also helps you to spot complexity details that you might miss by just doing an informal review.

Thanks to it's integration with TeamCity it made the decision easy for me as I rely a lot on catching   everything during the commit stage. If any rule is broken, the build agent will inform the developer so he/she can take ownership of the committed changes and there won't be any argument about it. All the rules are set as part of the project and shared as part of the solution. NDepend is about $335 per developer license and about $671 for a build server license. It is highly documented and I found all the help I needed online..the guys from NDepend did really a great job!.

Full Disclosure: I received a complimentary copy of NDepend to provide a review from my point of view. I'm a working professional with loads of years of commercial development experience with hundreds of hours (not to say millions) of code reviews on my shoulders and I don't have time to use useless tools and certainly this is not the case. NDepend is extremely helpful and a great addition to be considered by any development team out there.

Tuesday, 6 September 2016

Delphi REST client that talks to a REST Web API on a Raspberry PI in JavaScript

This article presents an approach for developing a mobile application to monitor a Raspberry Pi cluster using a simple Web API built in JavaScript. If you haven't followed my progress regarding the Raspberry Pi cluster you can read my article here. I would also encourage you to read the article regarding node.js and how I created a simple web monitor.

The main idea is to be able to check the status of my grid from anywhere and the best place for it is on my mobile device. In this case I extrapolated the idea from my initial article so I could reuse some of the concepts and leverage the User Interface with Delphi 10.1 Berlin. As I'm still waiting for the Delphi version that offers Linux Server support I will have to use a RESTful approach to communicate with my devices with ease.

In this case, the Android application will talk to the REST Web API that sits in one of the Raspberry PIs. This Web API is built using node.js. Here is the architecture of the example to build:

I have a cluster of 4 Raspberry Pi and I want to check if they are alive without having to go physically to them (I saw few times where those little ones were not responding properly after a reboot). Thanks to the RAD that Delphi 10.1 Berlin offers I can have my Android application up and running in less than one hour as the components are already in-built and I only have to worry about the design and the responsiveness of the final application.

On the back-end, I have a really simple JavaScript Web API built with node.js. This simple API awaits external requests through a controller named status which expects machine names (e.g. pi02:3000/status/pi01) or IP addresses (e.g. pi02:3000/status/ to test the connectivity to them. The result is in json format. The source code can be seen below:

Alternatively, the source code of the entire project can be found in Github at the link below:
In here you will find the Web API and the Delphi Application (Win64 and Android). You could also run it in iOS but this is up to you to test.

Here the Web API in action:
As you can see it's very simple. The API just pings the device you ask for and returns dead/alive in json format. This is then parsed by the client and represented in different statuses. To ping the devices I'm using the ping npm package.

The layout of my Android application using Delphi is as follows:

As you can see I'm using the components TRestClient, TRestRequest and TRestResponse. Now I need to be able to send REST requests to my WebAPI. To do this I need to configure these three components in the following way:

Place the three components in your form and they will automatically reference each other, then configure the TRestClient using the following parameters:

The important ones are Accept, AcceptCharset and BaseUrl (this last one will contain the url you want to request). Notice that this is just the base URL as the resource will be specified in the RestRequest component.
Now you only need to edit this "Resource" property and use the correct argument to check the machine you want to check.

To make sure that the application is fully responsive I'm using a different thread to check the communications and also show a nice TAniIndicator while the operation occurs. This will make the application fully responsive without freezing the app.

Here is a summary of the code for your review:

You can find the entire source code here:
Finally the application in action:

If you wonder :), I'm using Vysor Chrome app to mirror my android device and ScreenToGif to generate my Gifs.

You will find all the necessary details on my Github project (installation procedures on the Raspberry Pi, etc).

I look forward to your comments.

Sunday, 21 August 2016

Simple Website in Node.js for you Raspberry Pi 3

Now that I have a cluster of Raspberry Pi's my possibilities are endless. In this article I will show you how to create a simple website using Node.js with Express, Stylus and Pug

Node.js is is a platform built on Chrome's JavaScript runtime for easily building fast and scalable network applications, Express is a fast web framework for Node.js, Stylus is an innovative style-sheet language that compiles down to CSS and Pug is a succinct language for writing HTML templates.

To ease the pain of working with a Raspberry Pi, I will show you how to remote desktop it first for ease of use so you can code your website on the Pi itself without having to use the console.

The following operations have to be applied to every member of the cluster so Remote desktop is available to every node. The idea is to access each node via remote desktop using a laptop with windows 10.

Configure Remote desktop on the Raspberry Pi

The first thing to do is to update our version of Raspbian and ensure that all the packages are upgraded. This can be done with the following commands:

sudo apt-get update

Next, run the following command to upgrade any packages installed on your system that need upgrading:

sudo apt-get dist-upgrade

You'll have to perform this operation in all your cluster nodes. Now we are ready to install the packages we need: xrdp and samba. xrdp is an open source remote desktop protocol(rdp) server and Samba is the standard Windows interoperability suite of programs for Linux and Unix.

Run the following commands so we can remote desktop the Raspberry Pi's:

sudo apt-get -y install xrdp

The '-y' option will automatically answer yes to the default continue [Y/n] question.

Next step is to install the samba package so we will be able to access the Raspberry Pi's by its host name from Windows rather than by it’s IP address which changes as the node receives its IP address via DHCP:

sudo apt-get -y install samba

After the installation is successful, you should be able to ping the Raspberry Pi from your windows machine and perform the remote desktop:

If everything is configured correctly you should see the following screens:

To copy files from my Win10 machine to one of the nodes I use WinSCP. You can also create a shared folder on the Pi that's visible on your Win10 machine using Samba.

Now it's time to configure the rest.

Install Node.js

By default there is a pre-installed version of Node.js on the Raspberry Pi's. If you type node -v you'll see that the version of node.js is v0.10.29. We need to upgrade this version to a more recent one (v6.3.1 by the time I published this article).

Now our Raspberry Pi is ready for action. Let's see what are the next steps to create our Website.

To make things easier for you, I've created a project on Github that contains a sample website that you can use to start with. Installing the dependencies required through npm is a bit cumbersome so using a sample project makes things a bit easier.

Here is a screenshot of the site once it is up and running:

I'm using the site as a Raspberry Pi status monitor where there is a bit of javascript that pings each node on the grid. Then I use knockoutjs to bind the results to the page.

Once you've downloaded the repository, you only need to run the following commands to install the dependencies and run the website:
You can follow the instructions on my Github project:
The site in action:

Here is the list of installed packages for your reference:


Sunday, 14 August 2016

Raspberry Pi 3 Cluster Test

The following article describes a simple test that was executed on a 4 node (1 controller + 3 workers), Raspberry Pi cluster. The purpose is to obtain reproducible measures of MPI performance that can be useful to MPI developers.  If you haven't read my article about building a Raspberry Pi 3 cluster for parallel programming, you can find it here. The test is a matrix multiplication where each node will perform the calculations of a slice of it and send the results back to the main node. The test will play with 2 main variables: a) the size of the matrix and b) the number of nodes to use to perform the calculations. This should give us the time for each calculation and the speedup.

If you haven't seen my cluster yet, here is an image:

Test Description

The test consist of the following: 
The application generates two square (NxN) matrices A and B of a variable size and defined via arguments. Matrix B is by default visible to each node so we save time sending the array to each node. Then Matrix A is generated in the master node and sliced into several chunks and sent to each individual node of the cluster. The slicing is calculated in the master node. Once each individual node of the cluster has finalised with its calculations, they send the results back to the master node to combine the results and present the resultant matrix.

The slicing mechanism works as follows:

For the example above, imagine that we have a square matrix of size 6x6. We have 4 nodes in our cluster but only three of them are available for calculations. Node 0 or master is just there to arrange initial calculations, send the values to each node and then gather the results from each individual node and display results.

The architecture is quite simple but very common in these scenarios. The beauty of it is that we can increase the number of nodes in the cluster without having to change a single line of code in the application.

As we have a 6x6 matrix, we need to split that by the number of nodes available in the system. Notice that the size of the matrix needs to be divisible by the number of nodes available in the cluster. In this case we have 6 rows and 3 nodes, so there will be 2 rows of data for each node.


You can find all the source code and results on my github project:

In there you will find the source code of matrixmultiplication.py, the shell scripts that I used to run the tests, the logs and excel files that I used to gather all the details from each node.

The first step is to calculate the matrix multiplication using just 1 node and then see what's the speedup by using additional nodes.

The sizes of the matrices for this test are defined below:

  • 12x12
  • 60x60
  • 144x144
  • 216x216

Each matrix will be run against 3 nodes and from 1 to 4 cpus on each node. Every cycle of the application runs 10 times and we use the average value for defining our results.

Here are the results for the calculations above against 1 node:

Time is in seconds and we can see that the bigger the matrix, the longer it takes to be multiplied. Remember that the complexity for a matrix multiplication is O(n3). We can easily how the graphic tends to draw a cubic function. Just increasing the size of the matrix by 50% we increased the calculation time by 300%.

Here you can see the calculation that the application performs:
Let's see what happens when we run the same matrices against our cluster:

Matrix multiplication against 3 nodes (1 CPU each):

As expected we've reduced one third the execution time for our calculations.

Let's see what happens when we introduce more CPUs:

Matrix multiplication against 3 nodes (2 CPU each):

Matrix multiplication against 3 nodes (3 CPU each):

Matrix multiplication against 3 nodes (4 CPU each):

Notice that the RPI3 has 4 CPUs and we can control the number of CPU used through the machinefile and MPI. All the cpus are defined as a node in my machinefile and I made sure that each CPU was working while monitoring them. Below is a graph showing all four cpus working on one of my nodes while running the experiment 216x216 on 12 CPUs:

Here you can see an example running 3 CPUs on each PI. Notice how the CPU's reach 100% on each PI.

Here a sample script to grab the cpu usage for linux:
If we group the graphs together we have:

We can see that the highest throughput is achieved by splitting the matrix using as many nodes as possible and return the results back. Notice that time is not linear in this case as we would suppose to go down to 4s of calculations for each node but we go down to 8s instead (calculation of 216x216 against 12 cpus). We need to consider also that there is an overhead when running MPI and this needs to be taken into consideration. In any case the throughput can be seen in the following figure representing the speedup:

Using 4 CPUs per node gives the highest throughput with a speedup of 6.34. Speedup is calculated with the division of the SeqTime/ParaTime. With this configuration we achieve an 85% of time reduction for our calculations, allowing us to perform large calculations under seconds.

There are loads of tests still to perform on the cluster and this is just a simple example as to how to code a simple example into parallel computing. Please see my project on github for more info and reference.


Saturday, 30 July 2016

Creating a Raspberry Pi 3 Cluster - "Supercomputer", for parallel computing.

In this quick article I will show you how to create your own Raspberry Pi cluster for parallel computing via MPI (Messaging Passing Interface) library. This is a nice summer project now that I'm free from my Master's duties until September and I have been wanting to build this for a while. Thanks to the low prices of the Raspberry Pi we are now able to build this without spending too much. See below for the list of items you will need and price for the whole kit with 4 Pi's. 

The main decision behind this architecture is to choose which operating system and programming language to use to implement parallel computing. Because of my experience with HPC (High Performance Computing) and SGE (Sun Grid Engine) the best way to achieve this is by using either OpenMPI or MPICH3. These two are free open distributions, portable and very popular. As per the programming language, we have several alternatives: we could use c++, c#, python, etc. I could use winIoT for Rpi or simply a Linux distro. I'm a geek so I went for the latter as I do like interacting with command line interfaces, there is some beauty there that I can't explain :). 
So my decision is to use a Linux distribution as OS. In this case I'm choosing Raspbian Jessie which comes with some goodies installed by default and it will allow me to install all the components I need for my little project.

The second decision to make is to choose the programming language. In this case I'm choosing Python as I'm very familiar with it, it has plenty of libraries available and a nice integration with MPI via mpi4py library.

The other factor to take into account here is that I have two different models of RPi and I need to make sure that whatever I install on those will work well for both instances. I won't be able to install WinIoT to my old Rpi model A.

Building the cluster of Rpi's

The material that you will need is listed below with links included:

4 x Rpi 3 model B = 4 x £30 = £120
4 x 16Gb microSD card (Kingston) = 4 x £4.84 = £19.36
4 x USB to Micro USB Cable 0.5m = 4 x £0.88 = £3.5
1 x 5 port desktop switch = 1 x £6.49 = £6.49
5 x Ethernet patch cable 0.3m = 5 x £2.90 = £14.5
1 x USB Hub = 1 x £2.53 = £2.53

Total =  £192.38 (without considering delivery)

*This is a common configuration but you can start with just 2 or 3 RPi's and keep adding hardware later on.

Once all the components are assembled using the stackable case you should have something like the image below:

Below the image of my cluster up and running (see configuration section for more):

Configuring your cluster of RPi's

The idea is to configure one of the RPi's and then just clone the SD card and plug it to the next Rpi. Here you'll find a summary description of the steps to do to get you up and running:

Installing the OS

  • Download Raspbian Jessie image. I had some trouble downloading the zip file so I used the torrent link instead. See the version used below (4.4)
Once the OS image is downloaded, burn it to the SD card using Win32DiskImager:

Plug the microSD card to the first Pi (my PiController in my case) and power it up. Plug the Ethernet cable and head back to your computer to access the Pi remotely.

Open a command prompt (I'm using Win10 as my main computer) and type "ping raspberrypi". By default the Rpi's are named raspberrypi so they are easy to spot in your network. Once you ping it, you will be able to see the ip address of the device. Save this IP address for later as we will use it in PuTTY.

Launch PuTTY and type the IP address of the RaspberryPi:

You should see something similar to the image below:

login as: pi and password: raspberry (each Rpi uses same login/password)

Type: sudo raspi-config to configure our device:
  1. Go to Expand File System
  2. Go to Advanced Options -> HostName -> set it to PiController
  3. Go to Advanced Options -> MemorySplit -> set it to 16.
  4. Go to Advanced Options -> SSH -> Enable.
  5. Finish and leave the configuration.

Now we can start installing MPICH3 and MPI4PY. Notice that these steps take a while (> 4h) so arrange some free time for this beforehand:

Installing MPICH3

Follow the steps below to install version 3.2 of MPICH:
Once you've got everything installed you should see something like the image below:

Installing MPI4PY

Follow the steps below to install version 2.0 of MPI4PY:
once installed you should see something like the image below:

Now we have finished configuring the first RPi. Believe it or not if you reach this step and everything is working you should be proud of it. Now we will have to clone this SD card and place them into the other RPi's.

Preparing the other RPi's

As mentioned in the step above, bring the SD card to your main computer and save the content of the SD card using Win32DiskImager. Now copy this new image to the other SD cards. You should have now 4 SD cards with the same image. As now we have 4 cloned SD cards, my advice is to plug every Rpi individually and change the host name of every new added Rpi into the network, e.g. pi01, pi02, pi03, etc.

Do the following for every new RPi added into the network:


scan the network for a newly added device to find its IP address using a network scanner. Once found, use PuTTY to access it and use the commands below to set it up:

Type: sudo raspi-config to configure our device:
  1. Go to Expand File System
  2. Go to Advanced Options -> HostName -> set it to pi01
  3. Go to Advanced Options -> MemorySplit -> set it to 16.
  4. Go to Advanced Options -> SSH -> Enable.
  5. Finish and leave the configuration.
  6. sudo reboot.
Do the same for pi02 and pi03. Note that you can name your RPis the way you want.

Once done you should be able to see them all 4 using PuTTY:

Once completed, each Rpi will have its own IP. We need now to store each IP address into a host file also known as machinefile. This file contains the hosts which to start the processes on.

Go to your first RPi and type:

nano machinefile

and add the following IP addresses: (Note that you will have to add your own):
This will be used by the MPICH3 to communicate and send/receive messages between various nodes.

Configuring SSH keys for each RPi

Now we need to be able to command each RPi without using users/passwords. To do this we will have to generate SSH keys for each RPi and then share each key to each device under authorised devices. This is the way MPI will be able to talk to each device without worrying about credentials. This process is a bit tedious but once completed you will be able to run MPI without problems.

Run the following commands from the first Pi (PiController):
When running the ssh-keygen just hit enter (if you don't want to add specific passphrase) and the RSA key will be generated for you automatically.

Now we have configured the link between PiController to every single device but we still need to configure the other way around. So you will have to run the following commands from every individual device:

open the authorized_keys files and you will see the additional keys there. Each authorized_keys file on each device should contain 3 keys (as stated in the architecture diagram above).

Now the system is ready for testing.

Note that if your IP address changes, the keys will not be valid and the steps will have to be repeated. 

Testing the cluster

At this point I will just include a small example for you to test that the cluster works as expected. Later on I will publish a more complex scenario with a refined configuration to maximise the power of the cluster.

If everything is configured correctly, the following command should work correctly:

mpiexec -f machinefile -n 4 hostname

You can see that each Device has replied back and every key is used without problems.

Now run the following command to test a helloworld example:

mpiexec -f machinefile -n 4 python /home/pi/mpi4py-2.0.0/demo/helloworld.py

You should see something like the image below:

Now our system is ready to take any parallel computing application we want to develop.

Watch this space for more!.

Next steps

I will be creating more complex scenarios and squeezing the architecture to test its limits. Soon more!. Give it a go and let me know if you face any problem during the set up.