Tester toolbox 101 – Excel

There are three reasons I can think of that make Microsoft Excel one of the tools I use often enough to make it into my basic toolbox.

Creating the test data

Imagine testing an application that handles creating user accounts. You checked the happy path and a sample user was created. But can it handle a hundred? Ten thousand? You are an engineer so you start thinking about writing a script to create the test data. Writing scripts is fun, but scripting everything is not always the most efficient way to do stuff. Enter Excel and if your application supports CSV files all you have to do is just write first couple lines and then drag to create as much sample data you need. You can then use this data set in your automation, load testing, etc. etc.

[WPGP gif_id=”330″ width=”600″]

Excel will also create date ranges, long strings, random data… you name it.

Reporting

Now that’s an important part of every tester’s life. Typical reports requested from testers include testing progress report, product quality report, or bugs outstanding. And while some of reports needed are provided by the tools you already use, some specific ones may be missing. I can guarantee though that every Test Case Management System or Issue Tracking System will let you export data into a format that Excel can consume: directly into an XLS file, CSV, or via some sort of API. Last resort is using ODBC to talk to the TCM or ITS database directly.

I found the following Excel skills useful:

Your users use it too

This is obvious if your application allows for uploading Excel spreadsheets (to create graphs, to add users, to process orders etc.). Also if Excel shows up on the output (sales reports, current usage data, for example). But also if CSV files are used, your users are likely to create them or process them with Excel. So couple things to think about when it comes to CSV file processing:

  • When saving or opening a CSV file – Excel is likely to attempt using system locale-related ANSI codepage. So your application may have to set it correctly on the output, and process on the input.
  • But at least UTF8 should be always supported? Well… not on Mac.
  • CSV files are always comma-separated? Hint: they are not.

So here were my three reasons for using Excel. What are yours?

 

Source: http://www.giantbomb.com/steve-ballmer/3040-90975/images/
Winning at Testing with Excel

Piñata testing

Pinata

I did not come up with this term. We started using it back in 2009/2010 at Symantec inspired by Michael Bolton‘s  When do we stop testing? presentation. In it he mentions a Piñata heuristic as one of the ways to decide when you tested enough.

The idea is to beat on the Piñata (software) until candy (bugs) starts falling out. And stop when you hit the first dramatic problem.

This term also describes well a testing technique we were using at Symantec. Its basic principle is to use the Piñata beating approach to identify weak, or lower quality, parts of the software. You start testing the application, and when issues start showing up in certain area, you concentrate on it even more. Keep hammering as long as candy/bugs are falling out. Do it as long as the ratio of new issues is not slowing down. When that happens you take a pause.

Now comes the time to assess the damage made to the Piñata (our test subject):

  • Have you found something that may require a fix invalidating your further tests? If yes, you probably should stop here.
  • Do the issues found so far have something in common? Patterns showing up in bugs may indicate that there are repeated issues in the code under test. This might be a good time to chat with your developers(*). Maybe they are using a buggy library? Patterns may also show up in code coming from specific developer only. Maybe that person is not following coding practices others are using? If it seems like a more general problem, it might be worth testing other areas of your software for similar issues, not only the module that exposed them.
  • Looking at the issues, can you predict other issues showing up? For example: lack of good input handling may also signal potential SQL injection exposure. Think about scenarios like that one, and check the suspicious ones.
  • How does this specific area compare to the rest of your software? Is it significantly worse? Significantly better quality? Can you tell why is it better or worse?

Smashed donkey pinata on floor with candy

Depending on the results of the assessment described above you may decide to stop testing this specific area, or keep hitting it. If you keep testing, you will again stop and assess at some time, and decide you were done. You may eventually ask yourself one of two questions:

  1. Have I found all the issues in given area? Am I sure there is not even more critical issue still uncovered?
  2. Have I correctly identified the risky/weak part of the application?

The answer to both questions is “maybe not”. You may have not identified all the issues, there still may be severe bug hiding, and the other areas of your software may be even worse than this one. But when you look at testing as risk-reducing activity I think Piñata testing, when combined with other testing techniques, is a fun way to find good issues and great help in identifying weak spots.

 

(*) Actually any time is good to chat with developers. They are smart people and you can both learn from each other.

Why should Microsoft acquire BlackBerry next and also why they won’t do it

Disclaimer: I do currently work for MobileIron. This note, just like any other here, is by no means a view of my employer and is only my personal opinion.

I want to make my case for this: Microsoft acquires BlackBerry next. And also explain why it won’t happen, in my opinion.

Why yes:

  • Double the Microsoft mobile phone market share overnight
    Depends on how you count and who counts. And also where. If you count current shipments worldwide – Windows Phone ships more than Blackberry (IDC, August 2013). But if you count current U.S users – looks like BlackBerry still tops Windows Phone there (comScore, June 2013). Either way the differences are small, and typically within 1%-2% worldwide.
    Even with current Windows Phone growth – the changes may seem significant, like 100% year over year. But growing from 1,4% to 3,7% is easier than growing from 3,7% to 10% IDC predicts for Windows Phone in 2017. BlackBerry acquisition may drastically accelerate this growth.
  • Deliver the Windows phone Enterprise needs
    Regardless of what Microsoft people say – Windows Phone is not a phone for Enterprise. No VPN? Flimsy certificate support? Having a strong, Enterprise-oriented team could help add the missing features to Windows phones. Microsoft surely knows that large organizations need much more than just ActiveSync and BlackBerry/RIM has more than enough experience there.
    Also look at what Apple is delivering with iOS7. Apple will add a number of Enterprise-friendly features to their MDM protocol, which are lightyears ahead of what Microsoft MDM can currently do. Microsoft has a lot of catching up to do – BlackBerry acquisition would bring the necessary knowhow.
  • Support BlackBerry future to regain some of the market
    There are still a lot of corporate BB fans out there. But are they all jumping in joy and buying the next 10.000 BlackBerry 10 devices for their organization? They are not. They are waiting for what will happen to BlackBerry next. Some of them ran out of time and had to make the switch, just that they are not stuck with a dying platform. Some of them can still wait. And Microsoft acquisition could promise the stability they are waiting for. And secure their investments. Even if Microsoft’s decision is to phase out the products – at least that would be a planned operation.
  • Make an awesome device
    Windows Phone UI is loved by many. Combine that with Nokia hardware. Sprinkle with Enterprise features from BlackBerry. Just think about it.
  • Do it all cheap
    Current BlackBerry’s market capitalization is around $6B. Sure, it was 40% less same time last year. So Microsoft could do it now, or just wait a little more for the value to go down again. Looks to me BlackBerry’s stock price, just like Nokia for the last year, is partially sustained by rumors of acquisition. Maybe just wait for things to calm down after Nokia deal was announced and make a move later.

Why not:

  • Who is even left at BlackBerry
    If Microsoft were to buy BlackBerry for the talent – not sure how many people are left and whether it would still be worth it. Search the Huffington post, for example, for the lengthy list of layoffs at RIM/BlackBerry.
  • Microsoft does not want Enterprise
    Seems like the powers at Microsoft treat Windows Phone as primarily a consumer device. Maybe a BYOD. But never a device with Enterprise features first. And seeing what the WP8 updates are adding to the phone (still no VPN etc.) does not seem the consumer-first strategy is changing, and there is no place for BlackBerry in it.
  • Merging the platforms would be a pain
    How would you even do it? Run some kludgy virtualization of BlackBerry on Windows Phone? Probably not. Maybe let Windows Phones be managed through BES, and slowly phase out BB phones, as more apps become available for the surviving platform?
  • Why bother with hardware part of BlackBerry
    Microsoft already got itself the mobile phone manufacturer: Nokia. So why bother with the second one? You could maybe sell the hardware part of BB away, but who would buy it?
  • And, lastly, who would drive it from Microsoft side
    With Steven Ballmer announcing his leave, and Nokia acquisition, looks like there is more than plenty for Microsoft folks to worry about for the next 12 months. Who would bother with something like BlackBerry acquisition right now.

Take your iOS app UI apart with Reveal

One app that can make your life easier: Reveal

Last week had to investigate an app for potential UI issues. Had the Xcode project, unfortunately the specific part of the UI was built by a 3rd party library. Rather than debugging the code in place decided to give Reveal a try.

The integration was a breeze, pairing my Mac with the device also went without issues. All it took was 10 minutes of work to be able to see exactly what was happening in the part of the app’s UI I wanted to inspect. Loved the 3D view with layers – was very useful in finding the control used in the button that I needed to focus on.

If there is anything to complain about – that would be the performance and stability of the app. A full refresh of my app’s UI took a minute or two, and sometimes needed a manual refresh to pick up the updated app’s state. I was still impressed by how easy and useful this tool is.

They post some really nice screenshots on their website, so check them out:

http://revealapp.com/

 

Oh better Android e-mail client, where art thou? And this is where WP8 rocks.

One thing that I seriously miss after switching back to Android from Windows Phone 8 is a better email client. Don’t get me wrong – native email client is good, works fast, supports pretty much everything I need. But still it does not even come close to the speed and usability of the email client built into Windows phones.

If I were to name one feature missing in the native client that would be threaded view. Why Gmail client can get it, but not the native one? And I have tried things like k9mail – it is good. But it’s not as fast as the native one, and not even close to how fast and snappy Windows Phone 8 email was.

I am still glad that we have a bunch of decent alternatives on Android. Including stuff that my own MobileIron mentions:

  • MobileIron Android Email+
  • NitroDesk TouchDown
  • Email Apps on Samsung, HTC, and Motorola Devices

But maybe drop some of those 1000s of options (and I am looking at you k9mail) and focus on user experience instead?

The curious case of duplicated certificates sent by iOS

Here’s another one on certificates. Let’s say you are trying to do client certificate-based authentication in your iOS app and you came up with something like this:


- (void)connection:(NSURLConnection *)connection willSendRequestForAuthenticationChallenge:(NSURLAuthenticationChallenge *)challenge {
 if ([challenge previousFailureCount] > 0) {

 NSLog(@"Incorrect auth challenge %@", challenge);
 [[challenge sender] cancelAuthenticationChallenge:challenge];
 return;
 }

 // Checking the server certificate
 if ([[challenge protectionSpace] authenticationMethod] == NSURLAuthenticationMethodClientCertificate &&
 self.account.clientCertificate.privateRepresentation.length > 0) {

 /*
 Reading the certificate and creating the identity
 */

 NSData *p12Data = self.account.clientCertificate.privateRepresentation;

 CFStringRef password = (__bridge CFStringRef)(self.account.clientCertificate.privatePassword);
 const void *keys[] = { kSecImportExportPassphrase };
 const void *values[] = { password };
 CFDictionaryRef optionsDictionary = CFDictionaryCreate(NULL, keys, values, 1, NULL, NULL);
 CFArrayRef p12Items;

 OSStatus result = SecPKCS12Import((__bridge CFDataRef)p12Data, optionsDictionary, &p12Items);

 if(result == noErr) {
 CFDictionaryRef identityDict = CFArrayGetValueAtIndex(p12Items, 0);
 SecIdentityRef identityApp =(SecIdentityRef)CFDictionaryGetValue(identityDict,kSecImportItemIdentity);

 SecCertificateRef certRef;
 SecIdentityCopyCertificate(identityApp, &certRef);

 SecCertificateRef certArray[1] = { certRef };
 CFArrayRef myCerts = CFArrayCreate(NULL, (void *)certArray, 1, NULL);
 CFRelease(certRef);

 NSURLCredential *credential = [NSURLCredential credentialWithIdentity:identityApp certificates:(__bridge NSArray *)myCerts persistence:NSURLCredentialPersistencePermanent];
 CFRelease(myCerts);

 [[challenge sender] useCredential:credential forAuthenticationChallenge:challenge];
 }
 else {
 // Certificate is invalid or password is invalid given the certificate
 NSLog(@"Invalid certificate or password");
 return;
 }
 } else {
 // For normal authentication based on username and password. This could be NTLM or Default.
 if ([[challenge protectionSpace] authenticationMethod] == NSURLAuthenticationMethodClientCertificate) {
 NSLog(@"Certificate might be required");
 }
 NSURLCredential *credential = [NSURLCredential credentialWithUser:self.account.username
 password:self.account.password
 persistence:NSURLCredentialPersistenceNone];
 [[challenge sender] useCredential:credential forAuthenticationChallenge:challenge];
 }

}

All looks good, doesn’t it? And in most cases – it will even work fine. Unless your server handles the handshake, something goes wrong and you look at what happens on the wire:

iOS certs

Can you see what happens here? The client certificate is sent twice. That is not always an issue, but if your server is parsing the certificates, and expects them to be in order, and expects that the first one is the client certificate and what follows is the issuer… you are in trouble.

You may argue whether this is a client bug or the server issue… I would say the client should be fixed. Per RFC2246 and RFC5246 the certificate structure should contain sender’s cert once.

certificate_list
This is a sequence (chain) of X.509v3 certificates. The sender’s
certificate must come first in the list. Each following
certificate must directly certify the one preceding it. Because
certificate validation requires that root keys be distributed
independently, the self-signed certificate which specifies the
root certificate authority may optionally be omitted from the
chain, under the assumption that the remote end must already
possess it in order to validate it in any case.
The same message type and structure will be used for the client’s
response to a certificate request message. Note that a client may
send no certificates if it does not have an appropriate certificate
to send in response to the server’s authentication request.

So how do we fix it? Fortunately the fix is trivial. Change this line:

NSURLCredential *credential = [NSURLCredential credentialWithIdentity:identityApp certificates:(__bridge NSArray *)myCerts persistence:NSURLCredentialPersistencePermanent];

to this:

NSURLCredential *credential = [NSURLCredential credentialWithIdentity:identityApp certificates:nil persistence:NSURLCredentialPersistencePermanent];

And everybody is happy again. The app sends a single certificate only, no change on the server required.

Windows Phone 8: these are not the certificates you are looking for

tl;dr Looks like SSL stack on Windows Phone 8 is completely broken.

We were running some tests of WP8 devices against a server that requires client certificate-based authentication. After issuing a device certificate from our own CA – the device would authenticate against the server. Sometimes it would authenticate using our certificate, but sometimes it would show up with a completely different certificate.

Here are snippets of the SSL handshake. The interesting part starts when server sends a Certificate Request (click the image for full resolution):

Certificate Request

 

Our server issues a Certificate Request, with the request for the client to provide a certificate matching the Distinguished Name of our own local CA.

And here’s the response we see from the client:

So what happened here? The client responded with a certificate issued by Microsoft, rather than providing our certificate or failing the authentication.

The hierarchy of certificate is:

  • Certificate (id-at-commonName=urn:wp-ac-hash-2:PAzCfbUuekP_SrTA0NUecBjyqN1f5,id-at-organizationalUnitName=9DFF3EFECE1B1D3E352EF654DEFBB9DED7)
    • Certificate (id-at-commonName=Microsoft Genuine Windows Phone CA4,id-at-organizationalUnitName=GFS,id-at-organizationName=Microsoft Corporation,id-at-localityName=Redmond,id-at-stateOrProvinceName=WA,id-at-countryName=US)
      • Certificate (id-at-commonName=Microsoft Windows Phone PCA,id-at-organizationName=Microsoft Corporation,id-at-localityName=Redmond,id-at-stateOrProvinceName=Washington,id-at-countryName=US)

Google is not very helpful in explaining what “wp-ac-hash-2” is. But it looks to me like a device-specific certificate issued by Microsoft Global Foundation Services. Looking at http://www.globalfoundationservices.com/ – looks like this is the team that delivers Zune and other MS cloud services, so I would not be too surprised to see this certificate being used for WP8 store authentication or other MS services.

There are couple upsetting conclusions from seeing this Windows Phone 8 behavior:

  • WP8 may or may not be able to authenticate against your server if you need client certificate-based authentication
  • If there are multiple certificates on the device – phone may fail to reach out to Microsoft services, if they require client certificate. So I can imagine the Microsoft Store not being available, or phone not being able to verify software developer.
  • Your certificate is exposed. I think it may be possible now for a malicious user run an attack against services that rely on certificate-based authentication. All an attacker has to do is make the user access your website with her phone, attacker’s website would issue a Certificate Request hoping for the phone to respond with your certificate, and then proxy the request to your services. The traffic would look like issued by user’s device where in fact it would originate from malicious source.

Oh and did I mention that on top of that looks like certificates were an afterthought for Windows Phone 8? All you have to do is take a look at Windows Phone 8 Certificate Installation document. The only option of installing certificates being through IE or email? No MDM-installed certificates user can use?

 

 

FM radio coming in Windows Phone 8 GDR2?

The Verge quotes Microsoft sources claiming FM radio support in upcoming WP8 update. General Distribution Release 2 (GDR2, Portico was GDR1) is supposed to add FM radio support to Nokia Lumia 920, 820 and other compatible phones. Looks like the registry keys were there for a reason.

Verge also mentions Nokia-specific GDR2 updates: double-tap to wake the device from standby, flip to silence option, color profile settings for the device screen.

The update will be made available over the coming months, I assume before Google ultimately turns ActiveSync off for WP users on July 31st.

Peeking inside Windows Phone 8 registry

Windows Phone 8 lacks, unfortunately, a bunch of tools – including something that would show you logs or registry. This limits your ability to see what is really going on in the phone. Forums like Windows Phone 8 Development and Hacking show you how to mount ffu or describe partition layout – that helps.

So to aid you a bit see below for a way to read registry from a WP8 emulator

  1. Launch WP8 emulator and do whatever you want to check later (for example enroll the phone with Intune)
  2. While the emulator is running, copy %userprofile%\AppData\Local\Microsoft\XDE\*.avhd to another folder, for example on your Desktop (that folder should contain a single .avhd file). Why do you want to do it while the emulator is running? The reason is .avhd is not merged back into emulator’s disk when you shut it down. Every time you shut down the emulator it essentially resets all settings and forgets whatever you did in the previous session. There should be a way around this – by running XDE.exe manually – but since that stopped working on my system recently, I won’t consider it a working alternative.
  3. Rename .avhd to .vhd
  4. Right-click on the .vhd and mount the disk
  5. Your system now should have couple new drives mounted (WP8 partitions) – we’re looking for the one with \Windows\System32\config\SOFTWARE (my case it was G drive)
  6. Copy the entire \Windows\System32\config folder somewhere else, for example to your Desktop
  7. Clean file attributes (read-only, archive, system) on all files in this folder
  8. Mount SOFTWARE hive using regedit (http://www.petri.co.il/edit_registry_settings_for_users_other_than_myself.htm)
  9. Profit!

So what’s interesting there?

  • Looks like there are FM radio-related keys – so was Microsoft preparing to support it?
  • The MDM related stuff:
    • \Microsoft\Enrollment – phone enrollment information
    • \Microsoft\Enrollment\OmaDmRetry – OMA DM protocol settings
    • \Microsoft\EnterpriseAppManagement\Database\Tbl_EnrollmentToken – Application Enrollment Token (AET) information
    • \Microsoft\EnterpriseAppManagement\Database\Tbl_XAPRequest – Company Hub/Enterprise App information – including whether the download was successful and what was the error if it failed
    • \Microsoft\Provisioning – this is the goldmine, for example a list of supported CSPs, including hints that there is much more supported than what the official documentation mentions, like “BrowserFavorite” or “HotSpot”… both were there in Windows Mobile Oma