There are so many different techniques to test software. Sometimes it is difficult to decide what technique is most suitable. In part, the decision also depends on the software development phase. A techique that applies at unit test phase may not necessarily be suitable for acceptance testing.

Table 1 is a summary of the different techniques commonly used, mapped against testing phases. All applicable mappings are shaded in green. The table also gives the mapping of the roles of particular teams during the testing cycle. Definitions of different testing techniques can easily be found on many websites [1,2] and books [3,4].

Table 1: Testing Techniques and Roles Mapped against Testing Phases

Testing Techniques and Roles Mapped against Testing Phases

This mapping is based on my personal experience with testing. Every system is different. In some cases it may make sense to use a particular technique in a certain phase even if such a mapping is not listed in Table 1. For example, the table indicates the “Load & Stress Testing” does not apply to the integration phase. In some projects, it may make sense to do this during integration if the designer knows that the performance bottleneck for the system is at the interfaces.

Knowing correctly what to test – which dictates how to test – when indicates a certain maturity of the product team and management. It involves an understanding of the sytem, in and out. In involves anticipating things that could go wrong. It involves application of prior experience and collective knowledge building.

For example, it is easy to understand why “Usability Testing” should be performance during product acceptance but why would anyone want to do that during “Unit Testing”? Such a test performed at an earlier stage gives scope for user feedback and avoid expensive rework at a later stage. Another example is “Intrusive Testing” which is a dangerous activity during system integration and beyond, simply because the system could be delivered with the changes. If a similar test is needed at system integration phase, “Negative Testing” is better suited.

It will be seen that some activities span the entire test cycle. Regressing and test automation go hand-in-hand in many projects. The degree of automation and regression vary from project to project based on how much of present resources can be spared or how often the product undergoes changes.

Considering the test teams, it is seen in Table 1 that either development team or integration team may perform integration. The former is true of small organizations and the latter for bigger organizations. In general, there is no separate team for performing system integration. This is generally done by the test team straight after system testing.


  1. http://www.aptest.com/testtypes.html
  2. http://www.softwaretestinghelp.com/types-of-software-testing/
  3. John Watkins. Testing IT: An Off-the-Shelf Software Testing Process. Cambridge University Press. 2001.
  4. Publications of the Software Testing Institute.

Testing for Robustness

A couple of years ago I was tasked with writing a MAC decoder for HSDPA packets. Writing the decoder wasn’t that difficult but one of the requirements was to make it robust. What does it mean to make a decoder robust? In the simplest sense,

  • It should not crash even when given an input of rubbish.
  • It should identify errors and inform the user.
  • It should do a best-effort job of decoding as much of the input as makes sense.
  • It should exit gracefully even when configuration of HS MAC and PHY are null or invalid.

In the adopted implementation, the decoder would parse the input bit stream and decode it on the fly. It will flag errors as it encounters them. It will continue to decode as much of the input stream as possible and flag multiple errors when encountered. Naturally, to perform decoding of HSDPA packets, HSDPA configuration at MAC is a necessary control input to the decoder. In addition, we wanted to make the output user-friendly. We wanted to map the data stream to HS-SCCH, HS-DPCCH and HS-PDSCH configuration as well.

Once the decoder was coded, the next important task was to test it. Robustness of design is only as good as the tests on the design. It is customary to perform smoke tests for typical scenarios. Then there are boundary value tests. Then there are stress tests which are applicable more at the system level than at the module level. There are also performance tests, which was not a major concern for our platform.

Because the decoder parses configuration as well, it was important that the decoder considers the entire vector space of configuration as well.

The following possible decoding errors where identified:

  • genericError
  • hsDpcchConfigurationNull
  • hsPdschConfigurationNull
  • hsScchChannelInvalid
  • hsScchConfigurationNull
  • macConfigurationNull
  • numberOfMacDPdusOutofRange
  • queueIdentifierInvalid
  • selectedTfriIndexInvalid
  • sidInvalid
  • subFrameNumberOutofRange
  • tooManySidFields
  • transportBlockSizeIndexInvalid
  • transportBlockSizeIndexOutofRange
  • unexpectedEndOfStream
  • zeroMacDPdus
  • zeroSizedMacDPdus

Arriving at these possibilities requires a little bit of analysis of the Mac-hs PDU structure. One must look at all the fields, the range of valid values and the dependencies from one field to another. One must also look at all these in relation to the constraints imposed by the configuration.

Unit tests were created using BOOST. In particular, BOOST_AUTO_UNIT_TEST was used. This was already the case with most of the modules in the system. It’s so easy to use (like JUnit of Java) that it encourages developers to test their code as much as possible before releasing it for integration. If bugs should still remain undiscovered or creep in due to undocumented changes to interfaces, these unit tests can be expanded easily to test the bug fix. For some critical modules, we also had the practice of running these unit tests as part of the build process. This works well as an automated means of module verification even when the system is built on a different platform.

Below is a short list of tests created for the decoder:

  • allZeroData
  • hsPdschBasic
  • macDMultiplexing
  • multipleSID
  • multiplexingWithCtField
  • nonZeroDeltaPowerRcvNackReTx
  • nonZeroQidScch
  • nonZeroSubFrameCqiTsnHarqNewData
  • nonZeroTfri16Qam
  • nullConfiguration
  • randomData

It will be apparent that tests are not named as “test1”, “test2” and so forth. Just as function names and variable names ought to be descriptive, test names should indicate the purpose they serve. Note that each of the above tests can have many variations both in the encoded data input stream and the configuration of MAC and PHY. A test matrix is called for in these situations to figure out exactly what needs to be tested. However, when testing for robustness it makes sense to test each point of the matrix. Where the inputs are valid, decoding should be correct. Where they are invalid, expected errors should be captured.

In particular, let us consider the test name “randomData”. This runs an input stream of randomly generated bits (the stream itself is of random length) through the decoder. It does this for each possible configuration of MAC and PHY. The test is to see that the decoder does not crash. Randomness does not guarantee that there will be an error but it does make a valid test to ensure the decoder does not crash.

While specific tests gave me a great deal of confidence that the decoder worked correctly, it did not give me the same confidence about its robustness. It was only after the random data test that I discovered a few more bugs, fixed them and went a long way in making the decoder robust.

Data flow for Mac-hs Decoder Testing

Figure1: Data flow for Mac-hs Decoder Testing

I will conclude with a brief insight into the data flow during testing. This is illustrated in Figure 1. Let us note that,

  • ASN.1 is used as the input for all unit tests. ASN.1 is widely used in protocol engineering. It is the standard in 3GPP. It makes sense to use an input format that easy to read, reuse and maintain. Available tools (such as the already tested Converter) can be reused with confidence.
  • Converters are used to represent ASN.1 content as C++ objects.
  • Comparison between decoded PDU and expected PDU is done using C++ objects. A comparison operator can do this elegantly.
  • A third-party ASN.1 encoder is used to generate the encoded PDUs. This gives independence from the unit under test. An in-house encoder would not do. A bug in the latter could also be present in the decoder invalidating the test procedure.
  • It is important that every aspect of this test framework has already been tested in its own unit test framework. In this example, we should have prior confidence about the Converter and the Compare operator.

This morning I attended a two-hour presentation by Ankita Garg of IBM, Bangalore. The event was organized by ACM Bangalore at the premises of Computer Society of India (CSI), Bangalore Chapter on Infantry Road. It was about making Linux into an enterprise real-time OS. Here I present a summary of the talk. If you are more curious, you can download the complete presentation.


How does one define a real-time system? It’s all about guarantees made on latency and response times. Guarantees are made about an upper bound on the response time. For this to be possible the run-time behaviour of the system must be optimized by design. This leads to deterministic response times.

We have all heard of soft and hard real time systems. The difference between them is that the former is tolerant to occasional lapse in the guarantees while the latter is not. A hard real-time that doesn’t meet its deadline is said to have failed. If a system can meet its deadline 100% of the time then it can be formally called a hard real-time system.

Problems with Linux

Linux was not designed for real-time behaviour. For example, when a system call is made, kernel code is executed. Response time cannot be guaranteed because during its execution an interrupt could occur. This introduces non-determinism to response times.

The scheduling in Linux tries to achieve fairness while considering the priority of tasks. This is the case with priority based Round Robin scheduling. Scheduling is unpredictable because priorities are not always strict. A lower priority process could be running at times until the scheduler gets a chance to reschedule a higher priority process. Kernel is non-pre-emptive. When kernel is executing critical sections interrupts are disabled. One more problem is the resolution of timer which is at best 4 ms and usually 10 ms.

Solutions in the Real-Time Branch

A branch of development has been forked from the main branch. This is called CONFIG_PREEMPT_RT. This contains enhancements to support real-time requirements. Some of these enhancements have been ported to the main branch as well.

One important change is on spin locks. These are more like semaphores. Interrupts are not disabled so that these spin locks are pre-emptive. However, spin locks can be used in the old way as well. It all depends if the APIs are called for spinlock_t or raw_spinlock_t.

The sched_yield has a different behaviour too. A task that calls this is added back to the runnable queue but not at the end of the queue. Instead it is added to its right level of priority. This means that a lower priority process can face starvation. If such a thing does happen, it is only because design is faulty. Users need to consider setting correct priorities to their tasks. There is still the problem of priority inversion which is usually overcome using priority inheritance.

There is also the concept of push and pull. In a multiprocessor system, decision has to be made about the CPU where a task will run. A task waiting in the runnable queue of a particular CPU can be pushed or pulled to another CPU depending on tasks just completed.

Another area of change is IRQ. IRQ is kept simple while the processing is moved to an IRQ handler. There was some talk on soft IRQ, top half and bottom half, something I didn’t understand. I suppose these will be familiar to those who have worked on interrupt code on Linux.

In plain Linux, timers are based on the OS timer tick. This does not give high resolution. High resolution is achieved by using programmable interrupt timers, which requires support from hardware. Thus timers are separated from the OS timer ticks.

Futex is a new type of mutex that is fast if uncontested. It happens in the user space. Only if the mutex is busy it goes to kernel space and it takes the slower route.

In IBM, the speaker mentioned the tools she had used to tune the system for real-time: SystemTap, ftrace, oprofile, tuna

Proprietary Solutions

Other than what’s been discussed above, other solutions are available – RTLinux, L4Linux, Dual OS/Dual Core and using Virtual Logic, timesys… There was not a lot of discussion about these implementations.

Enterprise Real-Time

Why is real-time behaviour important for enterprises? This is because enterprises make guarantees through Service Level Agreements (SLA). They guarantee certain maximum delays which can only be achieved on an RTOS. The greater issue here is that such delays are not limited to the OS. Delays are as perceived by users. This means that delays at the application layer have to be considered too. It is easy to see that designers have to first address issues of real-time at the OS level before considering the same at the application layer.

The presenter gave application examples based on Java. Java, rather than C or C++, is more common for enterprise solutions these days than perhaps a decade ago. The problem with Java is that there are factors that introduce non-determinism:

  • Garbage collection: runs when system is low on free memory

  • Dynamic class loading: loads when the class is first accessed

  • Dynamic compilation: compiled once when required

Solutions exist for all these. For example, the garbage collector is made to run periodically when the application task is inactive. This makes response times more deterministic. Static class loading can be performed in advance. Instead of just-in-time (JIT) compilation, ahead-of-time compilation can be done – this replaces the usual *.jar files with *.jxe files. Some of these are part of IBM’s solution named WebSphere Real-Time.

There is wider community that is looking at RTSJ – Real-Time Specifications for Java.


Real-time guarantee is not just about the software. It is for the system as a whole. Hardware may provide certain functionality to enable real-time, as we have seen for the case of higher resolution of timers. Since real-time behaviour is about response times, in some cases performance may be compromised. Certain tasks may be slower but this is necessarily so because they there far more important tasks that need a time guarantee. There is indeed a trade-off between performance and real-time requirement. On average, real-time systems may not have much better response times than normal systems. However, building a real-time system is not about averages. It is about an absolute guarantee that is far difficult to meet on a non-real-time system.

Earlier today I attended the Wireless Test World 2008 at the Le Meridien, Bangalore, a day-long seminar presented by Agilent Technologies. My particular interest was in learning about the test solutions they had in place for WiMAX and LTE standards. The former is fairly mature in both its flavours – Fixed and Mobile. The latter is not only new but also incomplete.

Lots of items in LTE standardization are still for further study (FFS). As can be expected, Agilent’s solution for LTE is an initial offering to test early PHY layer implementations. A full-fledged solution that incorporates a validated Conformance Test suite for full stack testing is still sometime away. Core standards are getting ready right now. We can expect the first test specifications later this year. Something concrete can be expected on this front from Agilent at about the same time since they are involved closely in the standardization process. Building upon their existing partnership with Anite for the conformance of GSM/UMTS implementations, LTE conformance solution from Agilent will take a similar route.

The interest on the day was greater for WiMAX, arguably because more companies in India are working on it than on LTE. The immediate future may be more promising for WiMAX, but LTE stands an equal chance from 2011 and beyond.

The seminar was primarily presentations about established and emerging technologies, and the test capabilities Agilent offers for these. There was sufficient time in between to look at Agilent’s test solutions and see live demonstrations of their equipment. The keynote address by Mr Bhaktha Keshavachar was about the challenges faced in integrating wireless connectivity into mobile devices. In particular, the future is about bringing different radio standards into the laptop. WiFi and Bluetooth are already standard on almost all laptops. We expect the same with GPS, 3G, WiMAX and LTE in the coming years. Well, some of these models are already out in the market. We are not talking about external plugin modules but about MiniCards fully integrated into the laptop. Apparently, MiniCard are standardized and they have an extremely small form factor. Key challenges are the co-existence of multiple radio standards on the same device (interference), platform noise, high peak currents, overheating, inter-working, system performance, battery life… let’s be assured that there may challenges that we may not yet be aware of.

When it comes to WiMAX, Agilent has an impressive line-up of equipment to meet all needs of a WiMAX programme – R&D, Design Verification, Pre-Conformance, Conformance, Manufacturing, Network Deployment, Service Assurance. An OFDM signal analyzer was available as early as 2004 and a signal generator in 2005. A one-box solution was available in 2007 which today has Wave 2 and MIMO functionalities.

Agilent WiMAX Test Equipment

Agilent WiMAX Test Equipment

There are many hardware models with accompanying software packages which are sold as licensed options. These support standard WiMAX test requirements – signal generation, signal analysis, modulation analysis, constellation diagrams, power measurements, peak-to-average metrics, spectral flatness, ACLR measurement, CCDF measurement, SEM measurement. This includes support for OFDM, OFDMA and MIMO.

Small companies with limited budget would have to make choices. The availability of similar equipment under different model numbers can make it difficult to choose the right one. The best option is to talk to the sales team and get as much information as possible. It’s about knowing if a particular choice meets one’s requirement. It’s also about knowing if we are buying more than what we really need.

Based on my understanding, I have put together a subset of WiMAX test equipment from Agilent. This covers only equipment specific to WiMAX. Of course, there are lots of complementary equipment that can be used for any radio standard – power supplies, logic analyzers, oscilloscopes, battery drain analysis equipment and others.

Model Number



N5181A MXG

Vector signal generator

  • Upto 6GHz.

N5182A MXG

Vector signal generator

  • Upto 6GHz.

  • Has capability to be used with N7615B.

E4438C ESG

Vector signal generator

  • Has capability to be used with N7615B.

E8267D PSG

Vector signal generator

  • Has capability to be used with N7615B.


Signal Studio for 802.16 WiMAX

  • Software that can be used with vector signal generators.

  • Enables application specific signal generation without having to spend time in signal creation.

N9010A EXA

Signal Analyzer

  • Option 507 enables operation upto 7 GHz. Higher options are available.

  • Better value for money than MXA series.

  • Sophisticated user interface with colour coding of channels.

  • Provides enhanced spectrum analysis.

  • Provides support for measurement applications as optional extra – N9075A is for WiMAX.

  • Generally comes with 89600 series vector signal analysis software. Examples are 89601A and 89601X.

N9020A MXA

Signal Analyzer

  • Higher specs of N9010A. For example, has WiFi testing capability which its EXA counterpart doesn’t have.

Vector signal analysis measurement application

  • Can be used with N9010A EXA.


VXA signal analyzer measurement application

  • Can be used with N9010A EXA.


WiMAX measurement application

  • Can be used with signal analyzers N9010A and N9020A.

  • Enables WiMAX specific signal analysis.


Wireless Networking Test Set

  • One box solution with signal generator and analyzer.

  • Only for Mobile WiMAX.

  • Generally preferred over E6651A for manufacturing.

  • Used with N6301A.


WiMAX measurement application

  • Used for WiMAX transmitter testing.

  • Used with N8300A.


Mobile WiMAX Test Set

  • One box solution with signal generator and analyzer.

  • Only for Mobile WiMAX.

  • For R&D, pre-conformance and conformance testing.

  • For conformance testing, used with N6430A.

  • Used for Radiated Performance Test (RPT) by ETS-Lindgren.


WiMAX IEEE 802.16-2004/Cor2 D3 PCT

  • For Protocol Conformance Test (PCT).

  • Based on E6651A.

  • In partnership with Anite supplied software.

  • TTCN-3 runtime environment and interfaces from Testing Technologies.


WiMAX Wireless Test Manager

  • Software with ready-to-use tests.

  • Simplifies automated testing but not as formal as TTCN-3 based testing.

  • Can be used for pre-conformance testing.

  • Can be used with E6651A.

  • Can be used with E4438C ESG or N5182A MXG with N7615A/B license.

Note: It’s easy to find information about WiMAX on Agilent’s website. Go to URL http://www.agilent.com/find/wimax