Thursday, 13 December 2012

Capturing console output with Delphi 2010/XE (revised)

Following my previous post (Capturing console output with Delphi 2010/XE) and with all the great comments received on it, I have decided to publish the new solution provided by Lübbe Onken which solves the hanging issue when capturing the output for different kind of commands like ping, netstat, etc. The problem occurs on the last ReadFile which this solution will fix.
Here I'm summarizing Lübbe Onken comments on this issue:
"The current implementation assumes that if the external process is not finished yet, there must be something available to be read from the read pipe. This is not necessarily the case. If we use PeekNamedPipe to check for available data before entering the internal repeat loop, everything is fine.
I also put the CloseHandle into try ... finally and moved Application.ProcessMessages behind the internal read loop, because IMHO the screen update is better handled after processing the callback than before. But this is just cosmetic."

Here you can find the source code:


usage:

I want to say thanks to everyone who spend time looking at this issue and for making the community work and grow.
Jordi

Monday, 6 August 2012

Rpi and GPIO testing

In this article I'm showing my progress with the RPi using the GPIO. There are lots of interesting articles where people is doing the same, but you won't get the same feeling that I have until you test it by yourself. This application processor board is extremely delicate and you need the get used to its I/O, typical voltage values (3.3V), maximum current (8-11mA), etc. Few basic things that will help you not to burn the RPi (connect resistors after every LED, etc). My example is quite simple but with academical purposes and I'm sure it will inspire others. I'm using the slice of a pi extension board where you can easily get the connectors for GP0-GP7 outputs and lit some LED's programmatically.

Material (apart from your RPi):


First thing is to correctly identify the GP Inputs/Outputs from the Slice of Pi (GP0 to GP7 pins). Here are my findings and the correct pin numeration:
GPIO     Slice of Pi
Pin 11   GP0
Pin 12   GP1
Pin 13   GP2
Pin 15   GP3
Pin 16   GP4
Pin 18   GP5
Pin 22   GP6
Pin 07   GP7

As you can see, they are not in order, so you need to spend time looking for the right pins. I still need to identify all the other pins but I'll get to it.

The following code is just lighting up 8 LED's using 8 GPIO pins as outputs. Each pin is connected to a 220 ohm resistance plus a red LED and all terminations to GND.

Source code:


Example with two LED's:

Here is the video displaying the results:


Keep playing!.

Sunday, 29 July 2012

Raspberry Pi...the basics

After playing for a day with my Raspberry Pi, here I'm compiling a list of tools that would help you setting RPi up correctly. As I'm testing everything under Python and I want to control it using my laptop (Windows 7), I need different ways to be more productive and faster when developing on it. In my case I'll need a Python editor like geany, GPIO library and remote desktop using Xming and PuTTY via SSH. As stated in my previous post I'll build a small robot using the following mechanical and electronic parts (All of them bought from HobbyTronics) expending less than £50:
Update the system first:
root@raspberrypi:~# apt-get update
Installing Geany:
root@raspberrypi:~# apt-get install python geany xterm
Downloading GPIO RPi Python from Google code:
root@raspberrypi:~# wget https://raspberry-gpio-python.googlecode.com/files/python-rpi.gpio_0.3.1a-1_armhf.deb
--2012-07-29 10:20:19--
https://raspberry-gpio-python.googlecode.com/files/python-rpi.gpio_0.3.1a-1_armhf.deb
Resolving raspberry-gpio-python.googlecode.com
(raspberry-gpio-python.googlecode.com)... 173.194.66.82,
2a00:1450:400b:c00::52
Connecting to raspberry-gpio-python.googlecode.com
(raspberry-gpio-python.googlecode.com)|173.194.66.82|:443...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 10824 (11K) [application/x-debian-package]
Saving to: `python-rpi.gpio_0.3.1a-1_armhf.deb'

100%[======================================] 10,824      --.-K/s   in 0.1s

2012-07-29 10:20:27 (71.2 KB/s) - `python-rpi.gpio_0.3.1a-1_armhf.deb'
saved [10824/10824]
Installing the package GPIO RPi:
root@raspberrypi:~# sudo dpkg -i
python-rpi.gpio_0.3.1a-1_armhf.debSelecting previously unselected
package python-rpi.gpio.
(Reading database ... 54876 files and directories currently installed.)
Unpacking python-rpi.gpio (from python-rpi.gpio_0.3.1a-1_armhf.deb) ...
Setting up python-rpi.gpio (0.3.1a-1) ...
Testing the GPIO Rpi library:
Just open Geany and paste the following source code:

import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BOARD)

GPIO.setup(11, GPIO.IN)
GPIO.setup(12, GPIO.OUT)

#set up output 12 as true.

GPIO.output(12, GPIO.HIGH)

while True:
        time.sleep(1)
        input_11 = GPIO.input(11)
        print "%s %r" % ("Input 11 value is", input_11)
        time.sleep(1)

Compile it and run it. It should display the value of Pin 11. If you can change the state of that input, the display will show the change.

Starting up the SSH server:
root@raspberrypi:~# service ssh start

Download PuTTY and Xming and install them:
Once installed, run PuTTY and under Session options, use the IP of the RPi. (to get the IP just run the ifconfig command):

And enable the option X11 forwarding under connection options:

Once done, open the connection and use your credential to login (by default user: pi password: raspberry).

As soon as you are in, type startlxde and Xming will take over and it will display the RPi desktop on your local machine.


Check out the parameters for Xming as the short-cut should call: "...Xming.exe" -clipboard -rootless.

Related links:

Taking a bite of my Raspberry Pi!

Lots of thinks are going on, new interesting projects, new ideas but no time to post. I've just got my new Raspberry Pi and I'm planning to build a small robot with it using the GPIO. I'll use all the tools available from the internet and I'll see if I can write something using Pascal/Delphi and share it with the community (I've seen GPIO libraries for C# and Java). I'm really excited about it because it opens a set of possibilities, not only for Developers but also for students who will be hooked on to this beautiful board. In this post and the following ones you'll find my progress with my RPi and all the source code. I'm still building up the idea and I'm waiting for different parts of the robot to arrive. At least I've got the Raspberry Pi model B board, the keyboard, HDMI cable, an SD with Raspbian "weezy" distribution OS and loads of patience. 


Quick steps (Instructions working under Windows OS):

1. Download Raspbian.
2. Download Image writer for windows (to burn the OS into the SD Card).
3. Once installed, plug all necessary things to the board and turn it on!.
4. Configure the different options (SSH Server, Keyboard layout, Date&Time, etc)
5. run startx command and off you go!.


Running my "Hello world" using python:


Now I need to tinker with GPIO and get used to it. I'll use the GPIO.RPi python module to control the Input/Output signals.

Stay tuned for my forthcoming articles!.

Related links:

Saturday, 2 June 2012

Mocking methods

I have been using Mock objects a lot with Delphi and I have implemented a small example and that you can find available in my personal repository on Google code. This simple example plays with the signature of the methods and uses delegates to return what we want to return. There are lots of ways to mock objects and lots of really good available frameworks: PascalMock, Ultra Basic MockObjects, DelphiMocks, etc. but sometimes we just want to mock some of the methods reintroducing a new signature. This simple example has helped me a lot in my unit testing and I hope you find it enough interesting to use it.


Basically we need an Interface to Mock which will contain all the contracts of the instance. TMock will be mocking the Interface and we need to set up the methods on the mockable class in a particular way using delegates. With this method TMock will be able to inject a new delegate and use it in our unit testing.

The declaration of TMock is quite neat:

type
  IMock = interface
  end;

  TMock<T> = class(TInterfacedObject,IMock)
  private
    FMockClass : T;
  public
    function SetUp() : T;
    constructor Create(mockingClass : T);
  end;

{ TMock<T> }

constructor TMock<T>.Create(mockingClass: T);
begin
  FMockClass := mockingClass;
end;

function TMock<T>.SetUp(): T;
begin
  Result :=  FMockClass;
end;
Then the IServer interface will have two methods which I want to mock. The delegates on the IServer will have to be extended as a property to then be able to overwrite them during the unit testing.

  //Contract definition to be Mocked.
  IServer<T, TResult> = Interface
    //Send structure
    function GetSendMessage() : TFuncParam<T, TResult>;
    procedure SetSendMessage(const Value: TFuncParam<T, TResult>);
    property SendMessage : TFuncParam<T, TResult> read GetSendMessage write SetSendMessage;
    //Receive structure
    function GetReceiveMessage() : TFunc<T>;
    procedure SetReceiveMessage(const Value: TFunc<T>);
    property ReceiveMessage : TFunc<T> read GetReceiveMessage write SetReceiveMessage;
  End;

It is really important to define the visibility of the methods on the implementation side so the mock only sees the methods we really want to redefine.

Then when the mock test is defined, it will only display the methods available to inject:
To set up the test, we only need to use the following declaration:

procedure TestTProtocolServer.TestCommunicateWithMock;
var
  ReturnValue: Boolean;
  Server: IServer<String, Boolean>;
  mock : TMock<IServer<String, Boolean>>;
begin
  Server := TServer.Create;

  mock := TMock<IServer<String, Boolean>>.Create(Server);

  //Set up the mocking methods using delegates:
  mock.SetUp().SendMessage := (function (message : string) : boolean
  begin
    result := True; //Return always true whatever the message is
  end);

  mock.SetUp().ReceiveMessage := (function () : string
  begin
    //Return the same message the server would reply
    result := 'This is the message from the server!';
  end);

  FProtocolServer := TProtocolServer.Create(mock.SetUp());
  try
    ReturnValue := FProtocolServer.Communicate;
    CheckTrue(ReturnValue, 'Communication with the Server Failed');
  finally
    Server := nil;
    mock.Free;
    FProtocolServer.Free;
    FProtocolServer := nil;
  end;
end;

With this simple method we can reuse existing code and just rewrite what we need to mock as the method we are calling depends on other objects/libraries, etc which are not testable. Note that this is not a framework, it is just a simple example where I need to mock a method and I don't want to use any of the existing frameworks. I like the way moq works and I wanted something similar (I wish we had lambda expressions, maybe one day!):

var mock = new Mock<ITest>();

mock.Setup(p => p.SendMessage(Is.Any<String>))
    .Returns(true)

It has loads of drawbacks as you will need to change the signature of your methods using delegates and sometimes it is not easy to do it, but I have been using it and it does the job.

Please, have a look at the repository and do not hesitate to leave any comment.

Thursday, 31 May 2012

Project Updates

It is time for a retrospective now that we have gone through the first half of the year. Projects are coming along and I will be updating  TDPE (Thundax Delphi Physics Engine)  and  PZaggy (Graph tool)  which will have the source code available by the end of the year. I have been working on side projects like ATOM Monitor to keep track of your atom leaks (and yes, Delphi has atom leaks) and my personal repositories where you will find Delphi code snippets and small utilities I have been developing throughout the years and you will be able to download them and use them for free. This is the way to make Delphi community grow and improve the way we develop Delphi apps using the power of the language using: fluent interfaces, delegates, mock objects, Unit tests, etc.

I will be redesigning as well, the entire website to make it more functional with an emphasis on Open Source projects and other really interesting stuff.

So, stay tuned and I look forward to your comments!.
Jordi

Thursday, 5 April 2012

DUnit and TCustomAttributes

In this article I am playing with Unit tests and TCustomAttributes as I am working on different ideas to build up a lightweight testing framework.  Basically I liked the idea of TestCase attribute from NUnit and I wanted to do something similar using TCustomAttributes and accessing those attributes using the Delphi RTTI library. To understand better my purposes have a look at the following example extracted from NUnit webpage:

[TestCase(12,3,4)]
[TestCase(12,2,6)]
[TestCase(12,4,3)]
public void DivideTest(int n, int d, int q)
{
  Assert.AreEqual( q, n / d );
}

This simple example will execute three times the method or test using the parameters defined on the test case attributes. What if we could perform something similar? Would not be cool?

Here is what I have done so far:


Creation of the Custom Attributes:
type
  TUserPasswordAttribute = class(TCustomAttribute)
  private
    FPassword: string;
    FUserName: string;
    Fresponse: Boolean;
    procedure SetPassword(const Value: string);
    procedure SetUserName(const Value: string);
    procedure Setresponse(const Value: Boolean);
  public
    constructor Create(aUserName: string; aPassword: string; aResponse : Boolean);
    property UserName: string read FUserName write SetUserName;
    property Password: string read FPassword write SetPassword;
    property response : Boolean read Fresponse write Setresponse;
  end;

  TUserAgeAttribute = class(TCustomAttribute)
  private
    FAge: integer;
    FUserName: String;
    Fresponse: Boolean;
    procedure SetAge(const Value: integer);
    procedure SetUserName(const Value: String);
    procedure Setresponse(const Value: Boolean);
  public
    property UserName : String read FUserName write SetUserName;
    property Age : integer read FAge write SetAge;
    property response : Boolean read Fresponse write Setresponse;
    constructor Create(aUserName : string; aAge : Integer; aResponse : Boolean);
  end;

{ TUserPasswordAttribute }

constructor TUserPasswordAttribute.Create(aUserName, aPassword: string; aResponse : Boolean);
begin
  SetUserName(aUserName);
  SetPassword(aPassword);
  Setresponse(aResponse);
end;

procedure TUserPasswordAttribute.SetPassword(const Value: string);
begin
  FPassword := Value;
end;

procedure TUserPasswordAttribute.Setresponse(const Value: Boolean);
begin
  Fresponse := Value;
end;

procedure TUserPasswordAttribute.SetUserName(const Value: string);
begin
  FUserName := Value;
end;

{ TUserAgeAttribute }

constructor TUserAgeAttribute.Create(aUserName: string; aAge: Integer; aResponse : Boolean);
begin
  SetUserName(aUserName);
  SetAge(aAge);
  Setresponse(aResponse);
end;

procedure TUserAgeAttribute.SetAge(const Value: integer);
begin
  FAge := Value;
end;

procedure TUserAgeAttribute.Setresponse(const Value: Boolean);
begin
  Fresponse := Value;
end;

procedure TUserAgeAttribute.SetUserName(const Value: String);
begin
  FUserName := Value;
end;

Those two custom attributes will serve as an example for what I intend to do. I need to test a login and some data from a current user and those bespoke attributes would be used by the test case.

The Framework:
type
  TAttributeProc = reference to procedure(CustomAttr: TCustomAttribute);

  TFrameworkTestCase = class(TTestCase)
  public
    procedure TestAttributesMethod(CustomProc: TAttributeProc);
  end;

implementation

{ TFrameworkTestCase }

procedure TFrameworkTestCase.TestAttributesMethod(CustomProc: TAttributeProc);
var
  ContextRtti: TRttiContext;
  RttiType: TRttiType;
  RttiMethod: TRttiMethod;
  CustomAttr: TCustomAttribute;
begin
  ContextRtti := TRttiContext.Create;
  try
    RttiType := ContextRtti.GetType(Self.ClassType);
    for RttiMethod in RttiType.GetMethods do
      for CustomAttr in RttiMethod.GetAttributes do
        CustomProc(CustomAttr);
  finally
    ContextRtti.Free;
  end;
end;

As you can see TFrameworkTestCase inherits from TTestCase and it adds the magic of reading the custom attributes and invoking the delegate as many times needed.

Using the small framework:
unit TestUnit1;
{

  Delphi DUnit Test Case
  ----------------------
  This unit contains a skeleton test case class generated by the Test Case Wizard.
  Modify the generated code to correctly setup and call the methods from the unit
  being tested.

}

interface

uses
  TestFramework, Windows, Forms, Dialogs, Controls, Classes, RTTI, SysUtils, Variants,
  Graphics, Messages, Unit1, StdCtrls;

type
  TestTLoginCase = class(TFrameworkTestCase)
  strict private
    FLogin: TLogin;
  public
    procedure SetUp; override;
    procedure TearDown; override;
  published
    [TUserPasswordAttribute('User1', 'Password1', True)]
    [TUserPasswordAttribute('User2', 'Password2', True)]
    [TUserPasswordAttribute('User3', 'Password3', True)]
    [TUserPasswordAttribute('User3', '', False)]
    procedure TestUserLogin;
    [TUserAgeAttribute('User1', 26, True)]
    [TUserAgeAttribute('User2', 27, True)]
    [TUserAgeAttribute('User3', 28, False)]
    procedure TestUserAge;
  end;

implementation

procedure TestTLoginCase.SetUp;
begin
  FLogin := TLogin.Create;
end;

procedure TestTLoginCase.TearDown;
begin
  FLogin.Free;
  FLogin := nil;
end;

procedure TestTLoginCase.TestUserAge;
var
  aAge: integer;
  aUserName: string;
  aResponse : boolean;
begin
  TestAttributesMethod(procedure (CustomAttr : TCustomAttribute)
    begin
        if CustomAttr is TUserAgeAttribute then
        begin
          aUserName := TUserAgeAttribute(CustomAttr).UserName;
          aAge := TUserAgeAttribute(CustomAttr).Age;
          aResponse := TUserAgeAttribute(CustomAttr).Response;
          Assert(aResponse=(FLogin.fetchDatauser(aUserName)=aAge), 'Incorrect value ' + aUserName);
        end;
    end);
end;

procedure TestTLoginCase.TestUserLogin;
var
  aPassword: string;
  aUserName: string;
  aResponse : boolean;
begin
  TestAttributesMethod(procedure (CustomAttr : TCustomAttribute)
    begin
        if CustomAttr is TUserPasswordAttribute then
        begin
          aUserName := TUserPasswordAttribute(CustomAttr).UserName;
          aPassword := TUserPasswordAttribute(CustomAttr).Password;
          aResponse := TUserPasswordAttribute(CustomAttr).Response;
          Assert(FLogin.UserLogin(aUserName, aPassword)=aResponse, 'Incorrect user ' + aUserName);
        end;
    end);
end;

initialization
RegisterTest(TestTLoginCase.Suite);

end.

Notice that every test case contains a set of Custom attributes and they will be executed by using a delegate. I have included the Result parameter in the attribute so the test can know the result straight away and inform about it. 


Related links:

Sunday, 25 March 2012

Fluent Interfaces example using Delphi part II

Here is the second part of this interesting topic. As I'm still trying to redeem myself from my non popular first example, I'm sure this one will reach the expectations. For this example I'm trying to mimic the way LINQ works, using generics and delegates and I have adapted my solution using fluent Interfaces as well. This solution presents a IQueryList which contains a TList<T> which can be queried like we were using SQL. So, we can select certain values and apply a where clause to filter the final list. This example will give you a hint on how to correctly implement fluent interfaces and how to extend this functionality for your applications.


Delegates:
uses
  Generics.Collections, Generics.Defaults;

type
  TProc<T> = procedure (n : T) of Object;
  TProcList<T> = procedure (n : TList<T>) of Object;
  TFunc<T> = reference to function() : T;
  TFuncParam<T, TResult> = reference to function(param : T) : TResult;
  TFuncList<T, TResult> = reference to function(n : TList<T>) : TResult;
  TFuncListSelect<T, TResult> = reference to function(n : TList<T>) : TList<T>;

IQueryList:
IQueryList<T, TResult> = interface
    function Where(const param : TFuncParam<T, TResult>) : IQueryList<T, TResult>;
    function OrderBy(const AComparer: IComparer<T>) : IQueryList<T, TResult>;
    function Select(const param : TFuncListSelect<T, TResult>) : IQueryList<T, TResult>;
    function FillList(const param : TFuncList<T, TResult>) : IQueryList<T, TResult>;
    function Distinct() : IQueryList<T, TResult>;
    function List() : TList<T>;
  end;

  TQueryList<T, TResult> = class(TInterfacedObject, IQueryList<T, TResult>)
  private
    FList : TList<T>;
  protected
    function Where(const param : TFuncParam<T, TResult>) : IQueryList<T, TResult>;
    function FillList(const param : TFuncList<T, TResult>) : IQueryList<T, TResult>;
    function Select(const param : TFuncListSelect<T, TResult>) : IQueryList<T, TResult>;
    function Distinct() : IQueryList<T, TResult>;
    function List() : TList<T>;
    function OrderBy(const AComparer: IComparer<T>) : IQueryList<T, TResult>;
  public
    constructor Create();
    destructor Destroy(); override;
    class function New: IQueryList<T, TResult>;
  end;

constructor TQueryList<T, TResult>.Create();
begin
  FList := TList<T>.Create;
end;

destructor TQueryList<T, TResult>.Destroy;
begin
  if Assigned(FList) then
    FList.Free;
  inherited;
end;

function TQueryList<T, TResult>.Distinct: IQueryList<T, TResult>;
var
  list : TList<T>;
  i : integer;
begin
  list := TList<T>.Create();
  for i := 0 to FList.Count-1 do
  begin
    if not list.Contains(FList[i]) then
      list.Add(FList[i]);
  end;
  FList.Free;
  FList := list;
  result := Self;
end;

function TQueryList<T, TResult>.FillList(const param : TFuncList<T, TResult>): IQueryList<T, TResult>;
begin
  param(FList);
  Result := Self;
end;

function TQueryList<T, TResult>.List: TList<T>;
begin
  result := FList;
end;

class function TQueryList<T, TResult>.New: IQueryList<T, TResult>;
begin
  result := Create;
end;

function TQueryList<T, TResult>.OrderBy(const AComparer: IComparer<T>): IQueryList<T, TResult>;
begin
  FList.Sort(AComparer);
  result := Self;
end;

function TQueryList<T, TResult>.Select(const param: TFuncListSelect<T, TResult>): IQueryList<T, TResult>;
begin
  FList := param(FList);
  result := Self;
end;

function TQueryList<T, TResult>.Where(const param: TFuncParam<T, TResult>): IQueryList<T, TResult>;
var
  list : TList<T>;
  i: Integer;
  Comparer: IEqualityComparer<TResult>;
begin
  list := TList<T>.Create();
  for i := 0 to FList.Count-1 do
  begin
    Comparer := TEqualityComparer<TResult>.Default;
    if not Comparer.Equals(Default(TResult), param(FList[i])) then
      list.Add(FList[i]);
  end;
  FList.Free;
  FList := list;
  result := Self;
end;

Example Implementation <Integer, Boolean>:
procedure DisplayList();
var
  IqueryList : IQueryList<Integer, Boolean>;
  item : integer;
begin
  //Create the list and fill it up with random values
  IqueryList := TQueryList<Integer, Boolean>
    .New()
    .FillList(function ( list : TList<Integer> ) : Boolean
              var k : integer;
              begin
                for k := 0 to 100 do
                  list.Add(Random(100));
                result := true;
              end);

  //Display filtered values
  for item in IqueryList
    .Select(function ( list : TList<Integer> ) : TList<Integer>
              var
                k : integer;
                selectList : TList<Integer>;
              begin
                selectList := TList<Integer>.Create;
                for k := 0 to list.Count-1 do
                begin
                  if Abs(list.items[k]) > 0 then
                    selectList.Add(list.items[k]);
                end;
                list.Free;
                result := selectList;
              end)
    .Where(function ( i : integer) : Boolean
          begin
            result := (i > 50);
          end)
    .Where(function ( i : integer) : Boolean
          begin
            result := (i < 75);
          end)
    .OrderBy(TComparer<integer>.Construct(
         function (const L, R: integer): integer
         begin
           result := L - R; //Ascending
         end
     )).Distinct.List do
          WriteLn(IntToStr(item));
end;

This example fills up an initial list with 100 random numbers and then I query the list to give me all the values from the list which absolute value is greater than 0 and the values are between 50 and 75. From this list I want all the values ordered by value and I do not want repeated numbers (distinct method).

Example Implementation <String, String>:
procedure DisplayStrings();
const
   Chars = '1234567890ABCDEFGHJKLMNPQRSTUVWXYZ!';
var
  S: string;
  IqueryList : IQueryList<String, String>;
  item : String;
begin
  //Fill up the list with random strings
  IqueryList := TQueryList<String, String>
    .New
    .FillList(function ( list : TList<String> ) : String
              var
                k : integer;
                l : Integer;
              begin
                Randomize;
                for k := 0 to 100 do
                begin
                  S := '';
                  for l := 1 to 8 do
                    S := S + Chars[(Random(Length(Chars)) + 1)];
                  list.Add(S);
                end;
                result := '';
              end);
  //Query the list and retrieve all items which contains 'A'
  for item in IqueryList.Where(function ( i : string) : string
          begin
            if AnsiPos('A', i) > 0 then
              result := i;
          end).List do
    WriteLn(item);
end;

The string example is quite similar, it fills up a list with 100 random string values and then it filters the list displaying only items which contains the character "A".

Everything is based on interfaces so the garbage collector can step in and avoid memory leaks and you can find a sound widespread of generics, delegates and chaining methods all in once. This solution gives you control on the way data is treated and it can work with any type as it is using generics.

I have included all those examples in my personal repository on Google project. Please feel free to use it and  comment everything you like/dislike so we all can improve those examples. The Unit testing side includes interesting examples:

Enjoy it!.

I look forward to your comments.
Jordi
Related Links:

Saturday, 25 February 2012

Monitoring Global Atom Table part III

New version v1.4 has been released as there were few bugs detected. This version also includes a new and very interesting feature, inspecting atoms from windows services. "A Windows Service applications run in a different window station than the interactive station of the logged-on user. A window station is a secure object that contains a Clipboard, a set of global atoms, and a group of desktop objects. Because the station of the Windows service is not an interactive station, dialog boxes raised from within a Windows service application will not be seen and may cause your program to stop responding. Similarly, error messages should be logged in the Windows event log rather than raised in the user interface".
Source : Microsoft.
This actually means that a running service is using a different set of global atoms than the current user. To display those atoms, atom table monitor v1.4 includes an Atom scanner service which uses the same core engine than Atom monitor and retrieves the list of Global atoms and RWM atoms from the system under the window station.

Current version contains: Atom Table monitor v1.4.
- Atom monitor win32 stand-alone tool.
- List of common patterns.
- Atom scanner win32 service.
- Install / Unninstall service batch files.

Session selection screen:
If the service is up and running, we can select the option to display the atoms from the service session. If the service is not detected the monitor will stop itself.

Service session monitoring  RWM atoms:
This screen is displaying the amount of atoms which are being monitored by the service session. You can play with that by creating a small tool to leak atoms and use different configurations from the service. Have a look at my previous post How to run an application under active session account from a windows service.

User session monitoring RWM atoms:
Check out the amount of patterns which match an specific subset of atom strings. This will help you to rapidly identify which atoms are being created and which is the source.

Installing the service:
Use the batch files to install / uninstall ATOMScannerService.exe. Once installed, run it under local account.

Once up and running, select "Monitor Atoms from service session" on Option's tab and press scan atom table.

Related links:

How to run an application under active session account from a Windows service using Delphi

To run an application from a service impersonating an user account, first we need to install "JEDI API Library & Security Code Library" as it contains different interesting OS callings which are really useful in order to achieve our purposes. Once the library has been unzipped, create your own service where the library is located and add the following paths to your search path in your project options:

Then instead of using CreateProcess function to execute an application, we need to use CreateProcessAsUser function. The new process runs in the security context of the user represented by the specified token. The service must be run by the  LocalSystem account which is a predefined local account used by the service control manager. To be able to use the function from jedi-apilib which retrieves the token from the current user we need to use WTSQueryUserToken ( WtsGetActiveConsoleSessionID, hToken ) function. This function will only work under LocalSystem account which has SE_TCB_NAME property enabled, otherwise the query will be false.

Use the following example within your service:



Related links:

Monday, 20 February 2012

Monitoring Global Atom Table part II

Atom table monitor is now available on google code. Atom table is an important resource to take into account when developing win32/win64 applications as if the table gets depleted a "System Error. Code: 8. Not enough storage is available to process this command" would be returned leaving the system in an unresponsive / unstable state. This issue only happens under Windows Vista / 7 / Server 2008, so it is crucial to avoid leaking atoms. There is a very interesting article (Identifying atom leaks) from Microsoft debug team which actually shows the way to monitor the atom entries by debugging the kernel. In the first part of this article (Monitoring Global Atom Table part I) I had delved into detail by debugging the kernel and explaining how to display different atom entries. In that post I released alpha version of "Atom table monitor" which I have seen it is wrong as it shares the same memory area for global atom table and registerwindowmessage table. In this new version (v1.2) both tables are displayed into separate memory grids:

Global atom table:

RegisterWindowMessage table:

Display list of entries:

Matching string patterns:

Counters:

Testing screen:

Saturday, 18 February 2012

Use ADPlus to troubleshoot hangs and crashes in our Delphi applications

It's again debugging time and in this post I'm going to put forward how to use ADPlus to troubleshoot hangs or crashes of our Delphi applications. As I'm sure you know, applications are getting more and more complicated using different libraries and third party tools and sometimes it is quite difficult to find out where the current problem is located. It has happened to me quite often that a tool or app blows app without reason, without a proper exception (even though all exception handling mechanisms are there) so the final user never gets any indication about what's wrong. With few simple steps from this article you will be able to set up ADPlus correctly, attach it to your running process and then create the crash dump for its further analysis. Let's start with debugging time!.

Installing Debugging tools for windows:
This is the Swiss knife for any good developer. Bring it always with you as it is really helpful. You can download the library from here. Just install it and save it to and USB to be portable. Then spend time using it, not only using ADPlus but also with Windbg, as it is crucial that you know about how to analyse the crash dump.

Setting up our Delphi Applications:
To be sure our Delphi app is correctly identified, we need to generate the map file and then use an external tool to convert all that information to symbols.
Edit in the project options to generate debug information and a detailed map file. Then using map2dbg we will transform the map file into debug symbols (.dbg files).
Once your project is correctly set up, build it and you will get the map file. Download the latest version of map2dbg v1.3 and copy map2dbg.exe where your project is located and run the following command line:

C:\testAdPlus>map2dbg.exe Project1.exe
Converted 6882 symbols.

You will now see a .dbg file with all necessary information for ADPlus.

Setting up the Symbols:
Once again, if we run ADPlus without setting up the symbols, we will only see address of memory without descriptions:
Call stack below ---
*** ERROR: Module load completed but symbols could not be loaded for C:\testAdPlus\Project1.exe
 # ChildEBP RetAddr  Args to Child              
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0012f550 004832bd 01aba8c0 0045f20b 0012f700 Project1+0xb3295
01 0012f698 00487615 0012f91c 01aba8c0 0012f9a8 Project1+0x832bd
02 0012f6e4 0045eed1 01aba8c0 0012f91c 01aba8c0 Project1+0x87615
03 0012f710 00487768 00050980 0012f9a8 01acb650 Project1+0x5eed1
04 0012f86c 00487615 00000111 0012f91c 09010401 Project1+0x87768
05 0012f8b8 004a7c25 0012f9a8 00000111 01acb650 Project1+0x87615
06 0012f8e4 00486cb3 0012f8f8 00486ccb 0012f914 Project1+0xa7c25
07 0012f914 0043fd4a 00000111 00000980 00050980 Project1+0x86cb3
08 0012f92c 762cfd72 000209ae 00000111 00000980 Project1+0x3fd4a

Setting up _NT_SYMBOL_PATH environment variable:
Create a new general environment variable with the following name: _NT_SYMBOL_PATH and the following value:
symsrv*symsrv.dll*C:\Symbols*http://msdl.microsoft.com/download/symbols.
Where C:\Symbols is the path to your symbols. This is the same parameter set in WinDbg which I explained in my previous post (monitoring atom table part I).

Setting up ADPlus:
We can either run ADPlus using a simple configuration or use the configuration file which is much more complete. In any case, the fastest way is using the simple configuration which will help us to get the expected outcome.
C:\Program Files\Debugging Tools for Windows (x86)>adplus -crash -pmn Project1.exe -o C:\Adplus -mss c:\symbols

This configuration will run ADPlus looking for crashes, waiting for Project1.exe and it will output the results in C:\ADPlus and it will use the symbols from C:\Symbols.

Sample test:
This small piece of code will help me to generate a small access violation and then ADPlus will catch the crashing and it will generate the crash dump.

procedure TForm1.Button1Click(Sender: TObject);
begin
  try
    SimulateSystemException;
  finally
    Close;
  end;
end;

procedure TForm1.SimulateSystemException;
var
  p: PChar;
begin
  p := PChar(5);
  p^ := #9; //Access Violation here
end;

Once ADPlus is running, run your project, in my case Project1.exe and wait until it crashes (in my case it is just simulated so, the crash dump is instantly generated).

You will get a new line in the cmd window telling you:
Attaching to 4812 - Project1 in crash mode 02/18/2012 00:16:00

Once it crashes, go to C:\ADPlus folder and you will get a new folder with current date and the crash dump in it.

Analysing crash dump with Windbg:
Just open Windbg, load the first crash dump (FirstChance_Process_Shut_Down) and use the following commands:

0:000> !sym noisy
noisy mode - symbol prompts on
0:000> .reload Project1.exe
0:000> !analyze -v
ERROR: FindPlugIns 8007007b
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

DBGHELP: kernel32 - public symbols  c:\symbols\kernel32.pdb\FCCF6FAC09804D49A4BB256A77F519572\kernel32.pdb
DBGHELP: Project1.exe is stripped.  Searching for dbg file
SYMSRV:  c:\symbols\Project1.dbg\4F3D9543457000\Project1.dbg not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/Project1.dbg/4F3D9543457000/Project1.dbg not found
DBGHELP: .\Project1.dbg - file not found
DBGHELP: .\exe\Project1.dbg - path not found
DBGHELP: .\symbols\exe\Project1.dbg - path not found
DBGHELP: Project1.exe missing debug info.  Searching for pdb anyway
DBGHELP: Can't use symbol server for Project1.pdb - no header information available
DBGHELP: Project1.pdb - file not found
*** WARNING: Unable to verify checksum for Project1.exe
*** ERROR: Module load completed but symbols could not be loaded for Project1.exe
DBGHELP: Project1 - no symbols loaded
DBGHELP: user32 - public symbols c:\symbols\user32.pdb\CFD2C4C8EB9C406D8B6DC29512EB176A2\user32.pdb
DBGHELP: ole32 - public symbols c:\symbols\ole32.pdb\EDE30219D57144FAAC83675A6573D1982\ole32.pdb

Once processed, we can actually spot that the symbols are missing. This is because we need to place the symbols at the location defined by Windbg -> c:\symbols\Project1.dbg\4F3D9543457000\Project1.dbg. Just copy your dbg file into defined location and try again reloading Project1.exe from Windbg command line.

Now we can analyse again the crash dump and get all the information needed about the crash. This time with the correct symbols:

Call stack below ---
*** WARNING: Unable to verify checksum for C:\Users\jordi coll\Desktop\testAdPlus\Project1.exe
 # ChildEBP RetAddr  Args to Child              
00 0012f550 00473c9d 008fa8c0 00455747 0012f700 Project1!Unit1.TForm1.SimulateSystemException+0x5
01 0012f698 00477ff5 0012f91c 008fa8c0 0012f9a8 Project1!Controls.TControl.Click+0x75
02 0012f6e4 0045540d 008fa8c0 0012f91c 008fa8c0 Project1!Controls.TWinControl.WndProc+0x56d
03 0012f710 00478148 00010d2e 0012f9a8 0090b650 Project1!StdCtrls.TButtonControl.WndProc+0x71
04 0012f86c 00477ff5 00000111 0012f91c 09010401 Project1!Controls.DoControlMsg+0x28
05 0012f8b8 004984b1 0012f9a8 00000111 0090b650 Project1!Controls.TWinControl.WndProc+0x56d
06 0012f8e4 00477693 0012f8f8 004776ab 0012f914 Project1!Forms.TCustomForm.WndProc+0x599
07 0012f914 0043c146 00000111 00000d2e 00010d2e Project1!Controls.TWinControl.MainWndProc+0x2f
08 0012f92c 762cfd72 00010d26 00000111 00000d2e Project1!Classes.StdWndProc+0x16
09 0012f958 762cfe4a 00380fc8 00010d26 00000111 USER32!InternalCallWinProc+0x23
0a 0012f9d0 762d0943 00000000 00380fc8 00010d26 USER32!UserCallWinProcCheckWow+0x14b (FPO: [Non-Fpo])
0b 0012fa10 762d0b36 00fd2450 00fd23e8 00000d2e USER32!SendMessageWorker+0x4b7 (FPO: [Non-Fpo])
0c 0012fa30 74d1b4ba 00010d26 00000111 00000d2e USER32!SendMessageW+0x7c (FPO: [Non-Fpo])
0d 0012fa50 74d1b51c 00a54518 00000000 00080032 comctl32!Button_NotifyParent+0x3d (FPO: [Non-Fpo])
0e 0012fa6c 74d1b627 54010001 00000001 0012fb48 comctl32!Button_ReleaseCapture+0x112 (FPO: [Non-Fpo])
0f 0012facc 762cfd72 00010d2e 00000202 00000000 comctl32!Button_WndProc+0xa98 (FPO: [Non-Fpo])
10 0012faf8 762cfe4a 74cb70f8 00010d2e 00000202 USER32!InternalCallWinProc+0x23
11 0012fb70 762d09d3 00000000 74cb70f8 00010d2e USER32!UserCallWinProcCheckWow+0x14b (FPO: [Non-Fpo])
12 0012fba0 762d0979 74cb70f8 00010d2e 00000202 USER32!CallWindowProcAorW+0x97 (FPO: [Non-Fpo])
13 0012fbc0 004780f5 74cb70f8 00010d2e 00000202 USER32!CallWindowProcW+0x1b (FPO: [Non-Fpo])
14 0012fd44 00477ff5 00000202 008fa8c0 0012fd80 Project1!Controls.TWinControl.DefaultHandler+0xdd
15 0012fd90 0045540d 0012fe64 00000202 008fa8c0 Project1!Controls.TWinControl.WndProc+0x56d
16 0012fdd0 0043c146 00000202 00000000 00080032 Project1!StdCtrls.TButtonControl.WndProc+0x71
17 0012fde8 762cfd72 00010d2e 00000202 00000000 Project1!Classes.StdWndProc+0x16
18 0012fe14 762cfe4a 00380fbb 00010d2e 00000202 USER32!InternalCallWinProc+0x23
19 0012fe8c 762d018d 00000000 00380fbb 00010d2e USER32!UserCallWinProcCheckWow+0x14b (FPO: [Non-Fpo])
1a 0012fef0 762d022b 00380fbb 00000000 00010d2e USER32!DispatchMessageWorker+0x322 (FPO: [Non-Fpo])
1b 0012ff00 004a15d2 0012ff24 00010001 0012ff70 USER32!DispatchMessageW+0xf (FPO: [Non-Fpo])
1c 0012ff1c 004a1617 00010d2e 00000202 00000000 Project1!Forms.TApplication.ProcessMessage+0x122
1d 0012ff40 004a1942 0012ff54 004a194c 0012ff70 Project1!Forms.TApplication.HandleMessage+0xf
1e 0012ff70 004a9c96 0012ffc4 00405a14 0012ff88 Project1!Forms.TApplication.Run+0xce
1f 0012ff88 7610d0e9 7ffdd000 0012ffd4 77711603 Project1!Project1.Project1+0x4e
20 0012ff94 77711603 7ffdd000 7736d39e 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
21 0012ffd4 777115d6 004a9c48 7ffdd000 00000000 ntdll!__RtlUserThreadStart+0x23 (FPO: [Non-Fpo])
22 0012ffec 00000000 004a9c48 7ffdd000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

0:000> kvn
 # ChildEBP RetAddr  Args to Child              
00 0012ff30 774a5370 7747b148 ffffffff 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
01 0012ff34 7747b148 ffffffff 00000000 0012ff58 ntdll!ZwTerminateProcess+0xc (FPO: [2,0,0])
02 0012ff44 76f241ec 00000000 77e8f3b0 ffffffff ntdll!RtlExitUserProcess+0x7a (FPO: [Non-Fpo])
03 0012ff58 00405f73 00000000 0012ff88 00000000 kernel32!ExitProcess+0x12 (FPO: [1,0,0])
04 0012ff70 004a9c9b 0012ffc4 00405a14 0012ff88 Project1!System.Halt0+0xf3
05 0012ff88 76f2d0e9 7ffd3000 0012ffd4 77481603 Project1!Project1.Project1+0x53
06 0012ff94 77481603 7ffd3000 77aa9471 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
07 0012ffd4 774815d6 004a9c48 7ffd3000 00000000 ntdll!__RtlUserThreadStart+0x23 (FPO: [Non-Fpo])
08 0012ffec 00000000 004a9c48 7ffd3000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

The only remaining thing is to analyse the crash dump trying to identify which are the regions involved in the crash and tackle them!.

Happy ninja debugging!.

Related links:

Thursday, 9 February 2012

Monitoring Global Atom Table part I

The aim of this article is to give a sound understanding about Atoms, how to monitor them and check whether we have or have not any process that is potentially leaking atoms. As Microsoft very well defines:
"An atom table is a system-defined table that stores strings and corresponding identifiers. An application places a string in an atom table and receives a 16-bit integer, called an atom, that can be used to access the string. A string that has been placed in an atom table is called an atom name.
The system provides a number of atom tables. Each atom table serves a different purpose. Applications can use local atom tables to store their own item-name associations.
The system uses atom tables that are not directly accessible to applications. However, the application uses these atoms when calling a variety of functions. For example, registered clipboard formats are stored in an internal atom table used by the system. An application adds atoms to this atom table using the RegisterClipboardFormat function. Also, registered classes are stored in an internal atom table used by the system. An application adds atoms to this atom table using the RegisterClass or RegisterClassEx function."
source: Microsoft.

Atoms are stored as two-byte integers (uint16) and there can be 0xFFFF-0xC000=0x4000 (16384) entries maximum. If 0xFFFF is reached ERROR 8 is returned ("System Error. Code: 8. Not enough storage is available to process this command")

Discovering Atom table:
To display atom table we need to use Debugging tools for Windows: Windbg and debug the kernel to dump al registers.

First of all:
Install all necessary tools and then run Windbg and configure all symbol parameters to correctly use kernel debug mode(for windows 7/Server 2008 debug mode needs to be enforced using bcdedit /debug on ). The most important module here is win32k.sys.

Set Symbol file path (Menu File -> Symbol file path):
"symsrv*symsrv.dll*C:\Symbols*http://msdl.microsoft.com/download/symbols"

As we need win32k.sys, we need to generate the pdb files with symbols information using symchk tool from debugging directory:

C:\Program Files\Debugging Tools for Windows (x86)>symchk C:\temp\bb\win32k.sys /v
[SYMCHK] Searching for symbols to C:\temp\bb\win32k.sys in path symsrv*symsrv.dll*C:\Symbols*http://msdl.microsoft.com/download/symbols
DBGHELP: Symbol Search Path: symsrv*symsrv.dll*C:\Symbols*http://msdl.microsoft.com/download/symbols
[SYMCHK] Using search path "symsrv*symsrv.dll*C:\Symbols*http://msdl.microsoft.com/download/symbols"
DBGHELP: No header for C:\temp\bb\win32k.sys.  Searching for image on disk
DBGHELP: C:\temp\bb\win32k.sys - OK
SYMSRV:  win32k.pdb from http://msdl.microsoft.com/download/symbols: 1170455 bytes - copied
DBGHELP: win32k - public symbols
         C:\Symbols\win32k.pdb\D8788E4736B34ED5B8516DBB9D45E9942\win32k.pdb
[SYMCHK] MODULE64 Info ----------------------
[SYMCHK] Struct size: 1680 bytes
[SYMCHK] Base: 0xBF800000
[SYMCHK] Image size: 1839616 bytes
[SYMCHK] Date: 0x43446a58
[SYMCHK] Checksum: 0x001ca511
[SYMCHK] NumSyms: 0
[SYMCHK] SymType: SymPDB
[SYMCHK] ModName: win32k
[SYMCHK] ImageName: C:\temp\bb\win32k.sys
[SYMCHK] LoadedImage: C:\temp\bb\win32k.sys
[SYMCHK] PDB: "C:\Symbols\win32k.pdb\D8788E4736B34ED5B8516DBB9D45E9942\win32k.pdb"
[SYMCHK] CV: RSDS
[SYMCHK] CV DWORD: 0x53445352
[SYMCHK] CV Data:  win32k.pdb
[SYMCHK] PDB Sig:  0
[SYMCHK] PDB7 Sig: {D8788E47-36B3-4ED5-B851-6DBB9D45E994}
[SYMCHK] Age: 2
[SYMCHK] PDB Matched:  TRUE
[SYMCHK] DBG Matched:  TRUE
[SYMCHK] Line nubmers: FALSE
[SYMCHK] Global syms:  FALSE
[SYMCHK] Type Info:    TRUE
[SYMCHK] ------------------------------------
SymbolCheckVersion  0x00000002
Result              0x00130001
DbgFilename
DbgTimeDateStamp    0x43446a58
DbgSizeOfImage      0x001c1200
DbgChecksum         0x001ca511
PdbFilename         C:\Symbols\win32k.pdb\D8788E4736B34ED5B8516DBB9D45E9942\win32k.pdb
PdbSignature        {D8788E47-36B3-4ED5-B851-6DBB9D45E994}
PdbDbiAge           0x00000002
[SYMCHK] [ 0x00000000 - 0x00130001 ] Checked "C:\temp\bb\win32k.sys"

SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 1

Once done, you will see the pdb file generated in C:\Symbols folder and you would be able to load it from Windbg command line using the WinDbg commands http://windbg.info/doc/1-common-cmds.html.

kd> !sym noisy
noisy mode - symbol prompts on
kd> .reload win32k.sys
kd> lm
start    end        module name
804d7000 806cdc00   nt         (export symbols)       ntkrnlpa.exe
bf800000 bf9c1200   win32k     (deferred) 

Now we know that win32k.sys it's loaded in address bf800000 and the referenced memory can be displayed by using dq command (using a 64-bit pointer) and only the first dword (L1):

kd> dq win32k!UserAtomTableHandle L1
DBGHELP: win32k - public symbols  
         c:\symbols\win32k.pdb\D8788E4736B34ED5B8516DBB9D45E9942\win32k.pdb
bf9a7c18  81b4a270`e1c47230

kd> dq 81b4a270`e1c47230+10
e1c47240  e1a841c0`e1ba3008 e15fb8f8`e19fae00
e1c47250  e1ba3080`e1700738 e1546428`e1708f88
e1c47260  e15f4e28`e15f4c78 e19e9058`e1a04278
e1c47270  e15f9d20`e15fb878 e170eda8`e1704d90
e1c47280  e1a0bea8`e1f71658 e1a841f0`e1a61550
e1c47290  e15464a0`e1091388 e1b16bc8`e1542440
e1c472a0  e1001d00`e1a014c0 e181efe0`e1c42118
e1c472b0  e1bb9178`e16f33e0 e15fa858`e19e9078
kd> dq
e170076a  04080000`000081cc 04016d53`6d4d0001
e170077a  00006156`4d430c07 00260012`6b76002c
e170078a  0001007c`fec80000 7250ffff`00010000
e170079a  6f724773`7365636f 7963696c`6f507075
e17007aa  0407ffff`ffffffff d4987346`744e0c05
e17007ba  00000000`0000e16e 4fcf0000`00000000
e17007ca  29880003`00000000 04050000`0000e170
e17007da  0401e24e`4d430001 f6886944`624f0c02

Showing the content of the memory:
Now that we now where Global Atom table is allocated, we can start inspecting its elements by displaying Unicode characters from memory by using du command:
kd> du e1ba3080`e1700738+C
e1700744  "DDEMLUnicodeClient"
kd> du e1546428`e1708f88+C
e1708f94  "ACTIVATESHELLWINDOW"

We can dump the memory using 32-bit pointers (dd command) and get the same results:

kd> dd e1c47240
e1c47240  e1ba3008 e1a841c0 e19fae00 e15fb8f8
e1c47250  e1700738 e1ba3080 e1708f88 e1546428
e1c47260  e15f4c78 e15f4e28 e1a04278 e19e9058
e1c47270  e15fb878 e15f9d20 e1704d90 e170eda8
e1c47280  e1f71658 e1a0bea8 e1a61550 e1a841f0
e1c47290  e1091388 e15464a0 e1542440 e1b16bc8
e1c472a0  e1a014c0 e1001d00 e1c42118 e181efe0
e1c472b0  e16f33e0 e1bb9178 e19e9078 e15fa858
kd> dd e1c472b0
e1c472b0  e16f33e0 e1bb9178 e19e9078 e15fa858
e1c472c0  e1a84198 e1a61500 e195d558 e15ff980
e1c472d0  e15424a0 00000000 00000000 00000000
e1c472e0  00000000 00000000 00000000 00000000
e1c472f0  00000000 00000000 00000000 00000000
e1c47300  00000000 00000000 00000000 00000000
e1c47310  00000000 00000000 00000000 00000000
e1c47320  00000000 00000000 00000000 00000000
kd> du e1ba3008+c
e1ba3014  "Native"
kd> du e15fa858+c
e15fa864  "OleClipboardPersistOnFlush"

To display the whole content we can use the following function to loop through all buckets, listing all atoms created: 

kd> r $t0=poi(poi(win32k!UserAtomTableHandle)+C)
kd> .for(r $t1=0; @$t1<@$t0; r $t1=@$t1+1) {du poi(poi(win32k!UserAtomTableHandle)+10+(@$t1*4))+C}
e1ba3014  "Native"
e1a841cc  "ObjectLink"
e19fae0c  "6.0.2600.2180!tooltips_class32"
e15fb904  "Static"
e1700744  "DDEMLUnicodeClient"
e1ba308c  "DataObject"
e1708f94  "ACTIVATESHELLWINDOW"
e1546434  "FlashWState"
e15f4c84  "SysCH"
e15f4e34  "PBrush"
e1a04284  "6.0.2600.2180!msctls_progress32"
e19e9064  "SysIC"
e15fb884  "DDEMLEvent"
e15f9d2c  "SHELLHOOK"
e1704d9c  "Custom Link Source"
e170edb4  "ControlOfsM*EVJK"
e1f71664  "DesktopSFTBarHost"
e1a0beb4  "SysDT"
e1a6155c  "Link Source"
e1a841fc  "FileName"
e1091394  "ReBarWindow32"
e15464ac  "SysWNDO"
e154244c  "DDEMLAnsiServer"
e1b16bd4  "SysLink"
e1a014cc  "NetworkName"
e1001d0c  "USER32"
e1c42124  "OleDraw"
e181efec  "FileNameW"
e16f33ec  "MoreOlePrivateData"
e1bb9184  "Edit"
e19e9084  "Binary"
e15fa864  "OleClipboardPersistOnFlush"
e1a841a4  "OwnerLink"
e1a6150c  "ListBox"
e195d564  "Embed Source"
e15ff98c  "SysIMEL"
e15424ac  "ComboLBox"

(Source code from http://blog.lordjeb.com/tag/debugging/)
Another way of displaying the content of Global Atom Table is by using !gatom command from user session on Windbg. (http://msdn.microsoft.com/en-us/library/ff563166(v=vs.85).aspx)

0:001> !gatom
Global atom table c00c( 1) = Protocols (18) pinned
c00d( 1) = Topics (12) pinned
c00e( 1) = Formats (14) pinned
c01f(36) = OleEndPointID (26)
c026( 1) = AppProperties (26)
c024( 1) = Delphi00000DB8 (28)
c01e(94) = UxSubclassInfo (28)
c010( 1) = EditEnvItems (24) pinned
c022( 3) = CAddressComboEx_This (40)
c01d( 6) = OleDropTargetMarshalHwnd (48)
c028( 1) = Delphi000009C0 (28)
c008( 1) = StdDoVerbItem (26) pinned
c01c( 6) = OleDropTargetInterface (44)
c02c( 1) = WndProcPtr0040000000000DC4 (52)
c02b( 1) = ControlOfs0040000000000DC4 (52)
c014( 1) = Save (8) pinned
c016( 1) = MSDraw (12) pinned
c02a( 1) = WndProcPtr0040000000000940 (52)
c024( 1) = Delphi00000DB8 (28)
c01e(94) = UxSubclassInfo (28)
c023( 3) = CAutoComplete_This (36)
c007( 1) = StdShowItem (22) pinned
c011( 1) = True (8) pinned
c010( 1) = EditEnvItems (24) pinned
c019( 1) = PROGMAN (14)
c012( 1) = False (10) pinned
c015( 1) = Close (10) pinned
c022( 3) = CAddressComboEx_This (40)
c004( 1) = StdEditDocument (30) pinned
c01d( 6) = OleDropTargetMarshalHwnd (48)
c028( 1) = Delphi000009C0 (28)
c008( 1) = StdDoVerbItem (26) pinned
c01c( 6) = OleDropTargetInterface (44)
c02c( 1) = WndProcPtr0040000000000DC4 (52)
c005( 1) = StdNewfromTemplate (36) pinned
c02b( 1) = ControlOfs0040000000000DC4 (52)
c014( 1) = Save (8) pinned
c016( 1) = MSDraw (12) pinned
c02a( 1) = WndProcPtr0040000000000940 (52)
c002( 1) = StdNewDocument (28) pinned
c027( 1) = ddeconv (14)
c029( 1) = ControlOfs0040000000000940 (52)
c00f( 1) = Status (12) pinned
c017(27) = ThemePropScrollBarCtl (42)
c009( 1) = System (12) pinned
c01b( 3) = AnimationID (22)
c025( 1) = Folders (14)
c00b( 1) = StdDocumentName (30) pinned
c013( 1) = Change (12) pinned
c001( 1) = StdExit (14) pinned
c006( 1) = StdCloseDocument (32) pinned
c01a( 6) = CAddressBand_This (34)
c00a( 1) = OLEsystem (18) pinned
c018( 3) = CC32SubclassInfo (32)

But As we can see, we are not displaying all table completely as it is quite difficult to find the correct entry as the table contains atoms, registered clipboard formats, classes, etc. If we follow the steps from Microsoft debug team, the job is far way easy:

kd> dq win32k!UserAtomTableHandle l1
bf9a7c18  81c58ce0`e1aa1da8
lkd> dq 81c58ce0`e1aa1da8+10 l1
e1aa1db8  e1929b00`e1423c58
kd> dt nt!_HANDLE_TABLE e1929b00`e1423c58
   +0x000 TableCode        : 0xe19012b8
   +0x004 QuotaProcess     : 0xc0040004 _EPROCESS
   +0x008 UniqueProcessId  : 0x06010001 Void
   +0x00c HandleTableLock  : [4] _EX_PUSH_LOCK
   +0x01c HandleTableList  : _LIST_ENTRY [ 0x4d4d49 - 0xc060405 ]
   +0x024 HandleContentionEvent : _EX_PUSH_LOCK
   +0x028 DebugInfo        : (null) 
   +0x02c ExtraInfoPages   : 0n4390977
   +0x030 FirstFree        : 0x490050
   +0x034 LastFree         : 0x50005c
   +0x038 NextHandleNeedingPool : 0x50004e
   +0x03c HandleCount      : 0n3473456
   +0x040 Flags            : 0x310030
   +0x040 StrictFIFO       : 0y0
kd> !list "-t nt!_RTL_ATOM_TABLE_ENTRY.HashLink -e -x \"du @$extret+C\" e1929b00`e1423c58"
du @$extret+C 
e1423c64  "Native"
du @$extret+C 
e19012c4  "Object Descriptor"
du @$extret+C 
e1905f14  "Link Source Descriptor"
du @$extret+C 
e17e3fec  "Button"
du @$extret+C 
e108d014  "msctls_updown32"
du @$extret+C 
e108e224  "6.0.2600.2180!Static"
du @$extret+C 
e1c016f4  "MSIMEMouseOperation"
du @$extret+C 
e112b32c  "WorkerW"
du @$extret+C 
e167218c  "MSUIM.Msg.StubCleanUp"
du @$extret+C 
e1e6f12c  "Desktop More Programs Pane"
du @$extret+C 
e1e949fc  "FileContents"
du @$extret+C 
e1be4c1c  "CMBExecute"
du @$extret+C 
e2218d8c  "C:\WINDOWS\system32\xpsp2res.dll"
e2218dcc  ""
du @$extret+C 
e222a80c  "C:\WINDOWS\WinSxS\x86_Microsoft."
e222a84c  "Windows.Common-Controls_6595b641"
e222a88c  "44ccf1df_6.0.2600.2180_x-ww_a84f"
e222a8cc  "1ff9\comctl32.dll"
du @$extret+C 
e16524dc  "ControlOfs0040000000000DC4"
du @$extret+C 
e1971964  "ControlOfs0040000000000940"
du @$extret+C 
e214b4c4  "ControlOfs0040000000000EC4"
du @$extret+C 
e162340c  "ControlOfs0040000000000274"
du @$extret+C 
e1e7b7fc  "application/x-compressed"

Using Atom table monitor v1.0:
I have developed a small tool to visually monitor all atoms and look for different patterns on it using regular expressions and display the match with a different colour.

This small app will use GlobalGetAtomName (to get all atoms that have been added using GlobalAddAtom) and GetClipboardFormatName (to get all atoms that have been added using RegisterWindowMessage) functions to get all atoms and display them into a 128x128 memory grid using Delphi XE. It also keeps track of the amount of atoms through time plotting the results in a chart.

procedure ScanAtoms;
var
  index: WORD;
  arrAtom, arrClipboard: array [0 .. 1024] of char;
  AtomName, RWMName: string;
  lenAtom, lenClipboard: integer;
begin
  for index := $C000 to $FFFF do
  begin
    lenAtom := GlobalGetAtomName(index, arrAtom, 1024);
    lenClipboard := GetClipboardFormatName(index, arrClipboard, 1024);
    if lenAtom > 0 then
      FATomTable[index - $C000].atom := StrPas(arrAtom)
    else if lenClipboard > 0 then
      FATomTable[index - $C000].atom := StrPas(arrClipboard);
  end;
end;

Features:
  • Display global atom table from 0xC000 to 0xFFFF.
  • Display atom string from table.
  • Look for patterns and match them with a specific colour
  • Counters detailing the matches.
  • Plotting amount of atoms.
Testing:
Tool can be tested by using GlobalAddAtom or RegisterWindowMessages functions and check out the monitor.

Download:
Jordi Corbilla

Related links:

Tuesday, 31 January 2012

Projecting 3D points to a 2D screen coordinate system

In this article I will put forward a small function to project 3D points to a 2D coordinate system. I have used VLO framework to display all the points after the calculation of different common functions. The most difficult part is to correctly project 3D points into a 2D perspective. Basically the way to display the points is by using a viewport and setting it up correctly. Once all the points are calculated, we need to apply the transformations matrix to position the viewport and then draw the 3D object projected into a 2D canvas from the viewport perspective. The following function will use rotation matrix to correctly position the points. 

Sample code:
procedure T3DMesh.CalcPoints();
function d3Dto2D (viewport: T3DViewPort; point : T3DPoint; centre : T2DPoint; zoom : double) : T2DPoint;
var
  p : T3DPoint;
  t : T3DPoint;
  d2d : T2DPoint;
begin
  p.x := viewport.x + point.x;
  p.y := viewport.y + point.y;
  p.z := viewport.z + point.z;

  t.x := (x * cos(viewport.roll)) - (z * sin(viewport.roll));
  t.z := (x * sin(viewport.roll)) + (z * cos(viewport.roll));
  t.y := (y * cos(viewport.pitch)) - (t.z * sin(viewport.pitch));
  z := (t.y * cos(viewport.pitch)) - (t.z * sin(viewport.pitch));
  x := (t.x * cos(viewport.yaw)) - (t.y * sin(viewport.yaw));
  y := (t.x * sin(viewport.yaw)) + (t.y * cos(viewport.yaw));
  d2d := nil;
  if z > 0 then
  begin
      d2d := T2DPoint.Create(((x / z) * zoom) + centre.x, ((y / z) * zoom) + centre.y);
  end;
  result := d2d;
end ;

var
  listPoint : TObjectList;
  i: Integer;
  j: Integer;
  x, y, z: Double;
  d2dpoint : T2DPoint;
begin
  listPoint := TObjectList.Create;
  listPoint2 := TObjectList.Create;

  x :=-2.0;
  y := -2.0;
  z := 0.0;
  for i := 0 to 40 do
  begin
    y := -2.0;
    for j := 0 to 40 do
    begin
      z := cos((x * x + y * y) * 2) * exp(-1 * ((x * x) + (y * y)));
      listPoint.Add(T3DPoint.Create(x,y,z));
      y := y + 0.1;
    end;
    x := x + 0.1;
  end;

  for i := 0 to listPoint.count - 1 do
  begin
    d2dpoint := d3Dto2D(viewport, T3DPoint(listPoint[i]), centre, zoom);
    if Assigned(d2dpoint) then
      listPoint2.Add(d2dpoint);
  end;
  listPoint.Free;
end;


Examples:
z = sin(x) * cos (y):

z = cos((x^2+y^2)*2) * exp-(((x^2)+(y^2))):

z = x^2*exp(-x*x)*y^2*exp(-y*y):


With parameters:

Related links: