GUI automatic testing with XCtest and Xcode (2/2)

In part 1,  we saw some general ideas about automatic GUI testing, let’s now see the implementation and examples.

The files are on github , it contains the main files (GUITestSupport.h and .m) and some examples.

NOTE: this is very preliminary, and contributors are welcomed !

Let’s have a look at Example1 project. It is a very simple application, where user can enter two number in « a » and « b » fields, and the application displays the result « a+b »

Test setUp

[code language= »objc » highlight= »5″]
#import <Cocoa/Cocoa.h>
#import <XCTest/XCTest.h>
#import "GUITestSupport.h"

@interface Example1Tests : XCTestCaseGUI
@end

@implementation Example1Tests

– (void)setUp {
[super setUp];
// Put setup code here. This method is called before the invocation of each test method in the class.
}

– (void)tearDown {
// Put teardown code here. This method is called after the invocation of each test method in the class.
[super tearDown];
}
[/code]

GUITestSupport provides three new subclasses of XCTest :

  • XCTestGUI, which provides support for GUI testing. At setUp it will wait for the application to be active, and make the targeted window to be key window. You can override+ (BOOL) needActivate method to disable this behaviour ; override  – (NSString *) mainWindowTitle if you have several window in your application, to specify which window you are testing
  • XCTestCaseGUIDoc provides additional methods to handle Document based applications
  • XCTestCaseGUINewDoc is a subclass of XCTestCaseGUIDoc that will create a new, empty, document at setUp (thus for each test)
A first test

First test will simply put the two numbers and check the result

[code language= »objc » highlight= »2″]
– (void)test1 {
NSTextField *f1 = (NSTextField *)[self findViewOfClass:@"NSTextField" string:nil tag:1];
NSTextField *f2 = (NSTextField *)[self findViewOfClass:@"NSTextField" string:nil tag:2];
NSTextField *f3 = (NSTextField *)[self findViewOfClass:@"NSTextField" string:nil tag:3];
XCTAssert(f1);
XCTAssert(f2);
XCTAssert(f3);
[/code]

The first step is to find the controls in our window. Behind the scene, XCTestCaseGUI maintains a dictionary describing the view hierarchy. This dictionary can be obtained with – (NSDictionary *) testDescription, but we have a more convenient method :

[code language= »objc »]
– (NSView *)findViewOfClass:(NSString*)classname
string:(NSString *)str_or_nil
tag:(int)tag_or_minus1
[/code]

Classname is mandatory (and should be the exact class of the control or view, not a subclass), and filtering is done either on string (e.g. for a NSTextField) or on tag.
Note that in our app, we had to set different tags on each text field to enable testing

We then give focus on the « a » field, by clicking on it, and then we send keyboard stroke. Note that we start with a carriage return, which in NSTextField will select the content, so that we will replace it.

[code language= »objc » highlight= »2″]
[self sendMouseClickToControl:f1];[self sleep:0.1];
[self sendKeyboard:@"\n2\n" toControl:f1];
[self sleep:0.1];
[/code]

Also note the [self sleep:0.1]. It gives opportunity to the application to consume events and react to them.
- (void) sleep:(NSTimeInterval)duration is based on more generic method - (void) waitUntil:(BOOL(^)(void))cond which can wait for any arbitrary condition.

See below  for more trick about waitUntil: method

We then do the same for the « b » field:

[code language= »objc »]
[self sendMouseClickToControl:f2];[self sleep:0.1];
[self sendKeyboard:@"\n3\n" toControl:f2];
[self sleep:0.1];
[/code]

and check the result field :

[code language= »objc »]
NSString *s = [f3 stringValue];
XCTAssert([s isEqualToString:@"5"]);
}
[/code]

Note: depending on the test, it may be more convenient (and safe enough) to directly invoke method of your control rather than sending mouse and keyboard events : see for instance – (void) clickButton:(NSButton *)b method.

Finding views

The findViewOfClass:string:tag: method is not powerful in some cases, so we also have some new methods in NSView class :

[code language= »objc »]
@interface NSView (Tests)
– (NSView *) findSuperViewOfClass:(Class)c;
– (NSView *) findSubviewOfClass:(Class)c havingText:(NSString *)text;
– (NSView *) findSubviewOfClass:(Class)c havingTextPrefix:(NSString *)text;
@end
[/code]

For instance, in a view based NSTableView or NSOutlineView, you could :

  • find the NSTextField of a row, containing a given text, using  findViewOfClass:string:tag:
  • find the enclosing NSTableRowView using findSuperViewOfClass: ..
  • find other controls in this row using  findSubviewOfClass:..

Behind the scene

There is a trick on NSEvent consuming : all XCTest tests are run from a timer, and therefore the main loop would not consume GUI events like keystroke or mouse event. So running the maintop (eg. with  NSRunLoop runMode:.. or CFRunLoopRunInMode) would not be ok.

We must consume the events manually, using an ad hoc method :

[code language= »objc » highlight= »2″]
– (void) consumeEvents
{
// http://www.cocoabuilder.com/archive/cocoa/228473-receiving-user-events-from-within-an-nstimer-callback.html
NSEvent *event;
while ((event = [NSApp nextEventMatchingMask: NSAnyEventMask untilDate:nil inMode:NSEventTrackingRunLoopMode dequeue:YES])) {
[NSApp sendEvent: event];
}
}
[/code]

GUI automatic testing with XCtest and Xcode (1/2)

Xcode provides, with XCTest, a framework for automatic testing, both for unit test (testing part of the code) and for application test (test applied on the whole application). It is simple but efficient, but it do not cover GUI testing – e.g. simulating interaction between the user and your application.

This means, in a MVC architecture, that you can easily test the model, part of the controllers, and none of the view. In some (many) applications, where the model is a simple database processing, and most  of the complexity is in controllers and in GUI interaction, this means the coverage of the tests is really poor.

Automatic testing matters

Manual testing is not only error prone, but just not performed often enough – even if you’re working in a larger company, and have a dedicated test team, manual testing would need to be planned, etc..

Automated testing, assuming it is easy to run and fast, can be run very often by developers themselves, giving :

  • easy non regression tests
  • early regression detection
  • but the most important is : safe refactoring

You safely can deeply refactor part of your code if the code is covered by easy to run, high coverage test (and thus have safer, less complicated code, etc..). This is one of the reason why automatic testing is so important, and why it should be setup in very early stage of development.

Note that coverage is really important here – you can safely refactor a functionality if you know this functionality is indeed covered by your tests.

What to do with the GUI ?

There are several possible approaches for GUI testing, which you can find on web :

Manual testing

Needless to say, this is not my choice – but I believe it’s what most people do.

Check bindings

Some people proposed  to simply check that the correct bindings between views and controllers are indeed established. While incorrect bindings are a large error class in Cocoa application, this has a limited coverage, specially if you have a lot of processing in your controller class

Simulate user and check views

This obviously has the biggest coverage, but it raise two problems :

  • expansive test tools (e.g.: squish, eggplant) and complex (eg. Google’s Toolbox for Macintosh)
  • many tools would rely on screen shot comparaison

While screenshot comparaison is a good idea for large-scale production non regression tests,  it is not convenient during development phases, as the GUI look is very likely to be modified, and you don’t want to slow down the GUI polishing

A lightweight solution

I didn’t want to go with a complex framework ; XCTest is integrated with Xcode, it has a fast learning curve, it’s lightweight. I thus came to a custom solution that is

  • based on  XCTest – so all tests can be launched with a single CMD-U keystroke
  • lightweight : a single objective-C file
  • able to check view content (i.e. check that a given data is displayed) but being tolerant on how and where in the window it is displayed
  • able to simulate user inputs

Continue on part 2 (coming soon)

 

 

Print all sources to PDF from xcode

I’m no big fan of applescript – this is weird, strange syntax, etc… but being able to script most application is definitely great.

Here is a small – quick and dirty – script that will print all sources file to PDF in xcode. Note that xcode should have focus during all the execution

[code highlight= »4″]
tell application "Xcode"
set p to first project
set rootGroup to root group of p
set k to item references of rootGroup
repeat with x in k
set e to entire contents of x
repeat with fref in e
set g to full path of fref
log "path " & g
set l to leaf of fref
if l then
set f to file kind of fref
set g to full path of fref
set n to name of fref
open g
— display dialog g
tell application "System Events" to tell process "Xcode"
keystroke "p" using command down
delay 2 — or longer, if it takes longer
set w1 to window 1
set s to sheets of w1

set wk to first sheet of window 1
— display dialog wk

click menu button "PDF" of wk
click menu item "Save as PDF…" of menu 1 of menu button "PDF" of wk
delay 1 — or longer
–set ws to sheet 1 of window 1
–set ws2 to sheet 2 of window 1
set ws to sheet 1 of wk
keystroke (ASCII character 29)
— delay 1
keystroke "."
keystroke "p"
keystroke "d"
keystroke "f"
— delay 5
click button "Save" of ws — window "save"
delay 1 — or longer
end tell
else
log "skipped"
end if
— display dialog "alors?"
end repeat
end repeat
end tell
[/code]

Playing with Cappuccino (5)

In part 4, we built a basic address book web application.
Let’s add a filter to the table view – this may sound like Bells and whistles for a simple demo application, but it’s very convenient for the user, and require really no effort for us.

We’ll need a controller class to control the filtering, and since it is a small application, we will use the AppController class – in a larger application we may chose to add a custom class, and add an instance in interface builder.

Just add to AppController.j a CPPredicate instance variable :

[code language= »objc » highlight= »4″]
@implementation AppController : CPObject
{
@outlet CPWindow theWindow;
CPPredicate filterPredicate @accessors;

}
[/code]

Add a method to build a predicate whenever the search field value is changed :

[code language= »objc »]
– (IBAction) configArrayFilter:(id)sender
{

var searchString = [sender stringValue];
var predicate = nil;
if ([searchString length]) {
predicate = [CPPredicate predicateWithFormat:
@"(%K CONTAINS[cd] %@) OR (%K CONTAINS[cd] %@) OR (%K CONTAINS[cd] %@) OR (%K CONTAINS[cd] %@)",
@"name", searchString,
@"zipcode", searchString,
@"address", searchString,
@"town", searchString];
}
[self setFilterPredicate:predicate];
}
[/code]

See Apple document for predicate syntax details.

Connect the search field to this action.

Now all we need to do is to bind the array controller « Filter Predicate » property to our filterPredicate instance var :
capb9

We also need « add » and « delete » button, so just add two buttons, and connect them to the add: and the remove: method of the array controller.

Finally, we may want the new added value go be pre filled : to do so, we make a subclass of NSArrayController, named AddressArrayController :
[code language= »objc »]

@import <Foundation/Foundation.j>
@import <AppKit/AppKit.j>

@implementation AddressArrayController : CPArrayController

– (id)newObject
{
var n = [[CPMutableDictionary alloc]init];
[n setObject:@"unnamed" forKey:@"name"];
return n;
}

@end
[/code]
In interface builder, change the class of the ArrayController, and select the « automatically prepare content » option. Also, do not forget to add @import « AddressArrayController.j » in AppController to make sure the class is included in project.

You may also want to edit the menu – at least disable the entry that are meaningless or not implemented, and modify the Info.plist (application name, copyright notice…)

Your application should look like this now. Note that we only have around 110 lines of code…

 

Playing with Cappuccino (4)

In part 3, we defined the model part of our simple application. Let’s now do the GUI part – with virtually no code.

If XCodeCapp is running, you should notice that  a AddressBook.h file was added to your project, which is empty. It would contains only outlet and IBActions – just the definitions needed by interface builder, but our model has none for now.

Open the MainMenu.xib, and start adding AddressBook instance :

capb1

We will design our GUI using a « master+detail » approach – which is very usual on OS X and iOS : a master view gives list of all addresses, and a detail view gives more info on the selected address.

We thus add a table view, and some fields for the detail view. We also add a search field on top of the table view – in case we have many adresses, it is convenient to filter the table  :

capb3

And we’re done for the view part.  Notice that up to now, if we except the predefined address list, we have written very very few code.

It’s time to test : .cib file should be generated automatically by XCodeCapp, otherwise run nib2cib in you project directory, then run « jake debug », and  open the Build/Debug/CappuccinoExemple/index.html file with Firefox or another browser.

Do not use Safari to test, unless you open your generated files though a web server : Safari has some cache policies on file:/// url which are not ok for testing.

If you have some issues, use the index-debug.html file (cp index-debug.html Build/Debug/CappuccinoExemple/) – otherwise you won’t have information, and open error console in firefox.
(Note that having the console open slows down javascript a lot, so close it when you’re finished)

You probably get something like this :

capb4

Oh strange. But don’t panic, it’s easy to fix : in interface builder, select the box, and the « dimension » inspector : change the arrows so that the box is attached to top left :

capb5

Save it, do a « jake debug », and reload you’re page : it should look much better now. With dimensioning inspector, you can easily define how your window will appear when it’s resized.

It’s time to add the controller part. Cappuccino (similarly to Cocoa) provide a very convenient controller class CPArrayController (NSArrayController in cocoa), to display and handle arrays.

Just add it in interface builder, and bind the content to our model array.

capb6

Bind the column tables to the arrangedObjects of the array controller :

capb7

Bind each field to the ArrayController « selection » property :

capb8

run « nib2cib » (if XCodeCapp is not running) and « jake debug » , reload the page in your browser. If it does not work, uses index-debug.html and activate javascript console. You probably misspelled one of the binding.

So with virtually no code, we now have our master/detail view working, we can edit the data either in table view or in detail field. We can also sort the table view according to a column, etc… not bad!

In part 5, we will add some « add » and « delete » buttons, and make the search field work.

Playing with Cappuccino (3)

In part 2, we’ve setup a new Cappuccino project. It’s now time to start.

We’re going to build a simple application to store and display address book. The finished application is here.

Let’s start with model part. We won’t try to load/save address book on a server, or what so ever, we just want to do a simple demo.

We first define a AddressBook class:

[code language= »objc »]
@import <Foundation/Foundation.j>

@implementation AddressBook : CPObject {
CPMutableArray addresses @accessors;
}

@end
[/code]

Is that all ? Yes! An array that will hold our adresses. We will store each adress in a simple dictionary, so we don’t need any specific class for this.

Notice that we added the @accessors modifier. This is very similar to objective-c 2.0 properties : it will generate standard accessors for the instance variable.
Standard accessors for a variable named var are

[code language= »objc » gutter= »false »]
– (void)setVar:(type)v;
– (type) var;
[/code]

It is very important to use standard accessors, because KVO (key value observing) and KVC (key value coding), which are the fundamental mechanisms for bindings, rely on standard accessor. Shortly, KVO allows another class to be notified whenever someone change a given value using standard accessor, and to retrieve this value.
Google for « KVO » for more information, and for now just remember that we shall (most of the time) change value using standard accessors and not directly, to allows bindings.

We then add the usual initializer (note: in Objective-J, as in Objective-C, all instance variables are initialized to 0 by default, so we only need to initialize variables that should have other values). We don’t want to start the application with an empty address book, so we add a few adresses – three of the finest french restaurants.

[code language= »objc »]
– (id) init
{
self = [super init];
if (self) {
addresses = [[CPMutableArray alloc]init];
[addresses addObjectsFromArray:@[
@{ "name": "Bocuse",
"address" : "40 Quai de la Plage",
"town" : "Collonges au Mont d’Or",
"zipcode" : "69660" },
@{ "name": "Troigros",
"address" : "1 Place Jean Troisgros",
"town" : "Roanne",
"zipcode" : "42300" },
@{ "name": "La Côte Saint Jacques",
"address" : "14, Faubourg de Paris",
"town" : "Joigny",
"zipcode" : "89300" } ]];
}
return self;
}
[/code]

We use CPMutableArray – though in Objective-J it’s really the same as a CPArray – to emphasis that we will modified this array.

Finally, edit AppController.j and add

[code language= »objc »]
@import "AddressBook.j"
[/code]

Even if AppController does not need any info about AddressBook, it is mandatory, otherwise AddressBook class will not be included in the project.

That’s all for the code !

Check it compiles, by launching « jake debug » in a terminal, in the project directory :

[code gutter= »false »]
$ jake debug
(in …./CappuccinoExemple)
(in …./CappuccinoExemple)
—————————-
Debug app built at path: Build/Debug/CappuccinoExemple
—————————-
Calculating application file sizes…
Executables: 4511648, sprite data: 455078, total: 4966726
Build took 18 seconds
Build took 18 seconds
[/code]

Now let’s move to part 4 and build the GUI !

Playing with Cappuccino (2)

If you followed part 1, you are ready to build a new application.

Run XCodeCapp – it’s not in you’re Application folder, so either copy it in Application folder, or simply use spotlight to find and open it

spotlight

XCodeCapp only add a menu in the menu bar :

cap1

 

select « Create Project »

cap2

 

XCodeCapp creates a new project, launch xcode and opens the new project (Note that the project file itself is hidden in a .XcodeSupport
directory)

cap3

 

If you’re familiar with Objective-C and OS X (or iOS) programming, you’ll be in confort zone. You’ll find the usual AppController class – except it’s in Objective-J, and you’ll  notice that Objective-J classes have their Objective-C counterpart, which are empty : this is because Xcode interface builder only handles Objective-C, so the classes definitions and the outlet must be present in Objective-C. XCodeCapp will automatically build or update the Objective-C counterpart when  you modify an Objective-J file.

You’ll see that all Cappuccino classes have name starting with « CP » instead of « NS ». Basically, coming from Objective-C, if you change NS to CP, and remove pointers, you got Objective-J (there’s a little more differences, underlying language is Javascript, not C. See here for more details. You can jump to part 3 directly and start coding.

If you’re not familiar with Objective-C and Xcode, here are few clues :

Each application has an delegate class (here AppController), that control it’s behavior.

[code language= »objc »]
@implementation AppController : CPObject
{
@outlet CPWindow theWindow;
}

// methods

@end
[/code]

This is easy to read, we define and implement (there is no .h files in Objective-J, so interface is defined together with implementation) AppController as a subclass of CPObject. We define a single instance variable theWindow of class CPWindow. The @outlet modifier indicate that this instance variable can be set (or connected) through the interface builder.

We then have a few method implementations, e.g.:

[code language= »objc »]
– (void)applicationDidFinishLaunching:(CPNotification)aNotification
{
    // This is called when the application is done loading.
}
[/code]

Method implementation starts with a « – » – except class methods which start with a « + » (+/- similar to « static » methods in C++), followed by returned type, method name and arguments.
To invoke a method we use a syntax similar to Smalltalk, i.e. [object methodWith:arg1 and:arg2]

Method resolution is fully dynamic and done at runtime. Compared to C++, this would be similar (though more flexible) to having all method defined as virtual.

applicationDidFinishLaunching is a method that will be called by CPAppliation when launching is done – here we can add all our initialization.

Have a quick look to the MainMenu.xib file – the interface builder file.
cap4

it of course contains all visual elements  (menu, windows) but it also contains objects that will be instantiated when the .xib file is loaded (here: when the application is launched). On the right side, by selecting the « connection » inspector, you can see that the Application (the file owner)  connects the AppDelegate as its delegates, and that the outlet « theWindow » we had in AppDelegate is connected to the main window.

At launch time, not only the instances will be created, but the outlet instance variables will be initialized, referencing other objects created at the same time.

In addition to Objective-J presentation, you can find many tutorial on Objective-C and Cocoa : they mostly apply to Cappuccino.

Cocoa (and Cappuccino) application follow a MVC (model-view-controller) scheme, but you’ll see there is often very little or even no code for the controller part, because it is mostly handled by predefined classes, and by bindings.

You can now proceed to part 3 and start coding

Playing with Cappuccino (1)

Cappuccino is a unique Javascript framework to build applications. It is intended to give the look and feel of applications,  similar to Apple OS X interface – but wait ! not only the look and feel but also the programming. Yes, the whole cocoa framework (with few exceptions) is implemented in Objective-J (which is compiled in Javascript)

You can even use Xcode with interface builder to build your web application.

If you don’t have a mac, you can still use Cappuccino, but you’ll have to build the interface programmatically,  which (I believe) is a non-sense.

I would suggest to install Cappuccino from source, otherwise you won’t have XCodeCapp – a simple tool that helps integration with XCode. See here for instruction. This is the most painful part of using cappuccino ;  following the instruction carefully, and not changing the default path should help. Also restart from scratch if there is a download failure during the « bootstrap.sh »

Now, you are ready for your  first Cappuccino application. We’re going to build, in a step by step tutorial, a simple application to store and display address book. The finished application is here.