Archive for the ‘Software Engineering’ Category

There are so many different techniques to test software. Sometimes it is difficult to decide what technique is most suitable. In part, the decision also depends on the software development phase. A techique that applies at unit test phase may not necessarily be suitable for acceptance testing.

Table 1 is a summary of the different techniques commonly used, mapped against testing phases. All applicable mappings are shaded in green. The table also gives the mapping of the roles of particular teams during the testing cycle. Definitions of different testing techniques can easily be found on many websites [1,2] and books [3,4].

Table 1: Testing Techniques and Roles Mapped against Testing Phases

Testing Techniques and Roles Mapped against Testing Phases

This mapping is based on my personal experience with testing. Every system is different. In some cases it may make sense to use a particular technique in a certain phase even if such a mapping is not listed in Table 1. For example, the table indicates the “Load & Stress Testing” does not apply to the integration phase. In some projects, it may make sense to do this during integration if the designer knows that the performance bottleneck for the system is at the interfaces.

Knowing correctly what to test – which dictates how to test – when indicates a certain maturity of the product team and management. It involves an understanding of the sytem, in and out. In involves anticipating things that could go wrong. It involves application of prior experience and collective knowledge building.

For example, it is easy to understand why “Usability Testing” should be performance during product acceptance but why would anyone want to do that during “Unit Testing”? Such a test performed at an earlier stage gives scope for user feedback and avoid expensive rework at a later stage. Another example is “Intrusive Testing” which is a dangerous activity during system integration and beyond, simply because the system could be delivered with the changes. If a similar test is needed at system integration phase, “Negative Testing” is better suited.

It will be seen that some activities span the entire test cycle. Regressing and test automation go hand-in-hand in many projects. The degree of automation and regression vary from project to project based on how much of present resources can be spared or how often the product undergoes changes.

Considering the test teams, it is seen in Table 1 that either development team or integration team may perform integration. The former is true of small organizations and the latter for bigger organizations. In general, there is no separate team for performing system integration. This is generally done by the test team straight after system testing.


  1. http://www.aptest.com/testtypes.html
  2. http://www.softwaretestinghelp.com/types-of-software-testing/
  3. John Watkins. Testing IT: An Off-the-Shelf Software Testing Process. Cambridge University Press. 2001.
  4. Publications of the Software Testing Institute.

Read Full Post »

A couple of years ago I was tasked with writing a MAC decoder for HSDPA packets. Writing the decoder wasn’t that difficult but one of the requirements was to make it robust. What does it mean to make a decoder robust? In the simplest sense,

  • It should not crash even when given an input of rubbish.
  • It should identify errors and inform the user.
  • It should do a best-effort job of decoding as much of the input as makes sense.
  • It should exit gracefully even when configuration of HS MAC and PHY are null or invalid.

In the adopted implementation, the decoder would parse the input bit stream and decode it on the fly. It will flag errors as it encounters them. It will continue to decode as much of the input stream as possible and flag multiple errors when encountered. Naturally, to perform decoding of HSDPA packets, HSDPA configuration at MAC is a necessary control input to the decoder. In addition, we wanted to make the output user-friendly. We wanted to map the data stream to HS-SCCH, HS-DPCCH and HS-PDSCH configuration as well.

Once the decoder was coded, the next important task was to test it. Robustness of design is only as good as the tests on the design. It is customary to perform smoke tests for typical scenarios. Then there are boundary value tests. Then there are stress tests which are applicable more at the system level than at the module level. There are also performance tests, which was not a major concern for our platform.

Because the decoder parses configuration as well, it was important that the decoder considers the entire vector space of configuration as well.

The following possible decoding errors where identified:

  • genericError
  • hsDpcchConfigurationNull
  • hsPdschConfigurationNull
  • hsScchChannelInvalid
  • hsScchConfigurationNull
  • macConfigurationNull
  • numberOfMacDPdusOutofRange
  • queueIdentifierInvalid
  • selectedTfriIndexInvalid
  • sidInvalid
  • subFrameNumberOutofRange
  • tooManySidFields
  • transportBlockSizeIndexInvalid
  • transportBlockSizeIndexOutofRange
  • unexpectedEndOfStream
  • zeroMacDPdus
  • zeroSizedMacDPdus

Arriving at these possibilities requires a little bit of analysis of the Mac-hs PDU structure. One must look at all the fields, the range of valid values and the dependencies from one field to another. One must also look at all these in relation to the constraints imposed by the configuration.

Unit tests were created using BOOST. In particular, BOOST_AUTO_UNIT_TEST was used. This was already the case with most of the modules in the system. It’s so easy to use (like JUnit of Java) that it encourages developers to test their code as much as possible before releasing it for integration. If bugs should still remain undiscovered or creep in due to undocumented changes to interfaces, these unit tests can be expanded easily to test the bug fix. For some critical modules, we also had the practice of running these unit tests as part of the build process. This works well as an automated means of module verification even when the system is built on a different platform.

Below is a short list of tests created for the decoder:

  • allZeroData
  • hsPdschBasic
  • macDMultiplexing
  • multipleSID
  • multiplexingWithCtField
  • nonZeroDeltaPowerRcvNackReTx
  • nonZeroQidScch
  • nonZeroSubFrameCqiTsnHarqNewData
  • nonZeroTfri16Qam
  • nullConfiguration
  • randomData

It will be apparent that tests are not named as “test1”, “test2” and so forth. Just as function names and variable names ought to be descriptive, test names should indicate the purpose they serve. Note that each of the above tests can have many variations both in the encoded data input stream and the configuration of MAC and PHY. A test matrix is called for in these situations to figure out exactly what needs to be tested. However, when testing for robustness it makes sense to test each point of the matrix. Where the inputs are valid, decoding should be correct. Where they are invalid, expected errors should be captured.

In particular, let us consider the test name “randomData”. This runs an input stream of randomly generated bits (the stream itself is of random length) through the decoder. It does this for each possible configuration of MAC and PHY. The test is to see that the decoder does not crash. Randomness does not guarantee that there will be an error but it does make a valid test to ensure the decoder does not crash.

While specific tests gave me a great deal of confidence that the decoder worked correctly, it did not give me the same confidence about its robustness. It was only after the random data test that I discovered a few more bugs, fixed them and went a long way in making the decoder robust.

Data flow for Mac-hs Decoder Testing

Figure1: Data flow for Mac-hs Decoder Testing

I will conclude with a brief insight into the data flow during testing. This is illustrated in Figure 1. Let us note that,

  • ASN.1 is used as the input for all unit tests. ASN.1 is widely used in protocol engineering. It is the standard in 3GPP. It makes sense to use an input format that easy to read, reuse and maintain. Available tools (such as the already tested Converter) can be reused with confidence.
  • Converters are used to represent ASN.1 content as C++ objects.
  • Comparison between decoded PDU and expected PDU is done using C++ objects. A comparison operator can do this elegantly.
  • A third-party ASN.1 encoder is used to generate the encoded PDUs. This gives independence from the unit under test. An in-house encoder would not do. A bug in the latter could also be present in the decoder invalidating the test procedure.
  • It is important that every aspect of this test framework has already been tested in its own unit test framework. In this example, we should have prior confidence about the Converter and the Compare operator.

Read Full Post »

With the coming of age of packet-switched technology in preference to circuit-switched connections, many network systems are already based on soft-switch architecture. The fundamental concept in this architecture is that switching happens mostly in software. The role of software was previously minimal because switching used to happen in hardware. Connection circuits used to be established at the start of a call and torn down at call termination. With soft-switch architecture, the role of software is much more. Software therefore needs to be much more robust. Creating robust software involves many things but before that designers need to understand the requirements of such a software.

Yesterday at the British Council Library in Bangalore I picked up a wonderful book: Robust Communications Software by Greg Utas, published by John Wiley & Sons. The bulk of the book deals with the questions of “how” but the first chapter outlines the requirements of a carrier-grade system. I summarize these requirements with thoughts of my own.

Figure 1 shows the five basic attributes of a carrier-grade system. Although all are important in their own way, they are not necessarily of the same level of importance. It may be the case that reliability is more important that scalability. It may be that having a highly scalable system may require some compromise in terms of capacity. Some of these issues will be explained later in this post.

Figure 1: Requirements of a Carrier-Grade SystemCarrier grade system

While I was working in the UK, one morning the office phone system was down for about a couple of hours. During that time, there was no way for customers to reach either the support staff or the sales staff. The loss to the business was somewhat mitigated because some customers had the mobile contacts of our staff. Internally, e-mails served well during that interim period. I don’t know if we had a softswitch architecture in place. Either way, availability means that public communication sytems should be operational 24/7 all days of the week. Imagine someone trying to place an emergency call only to find that the system is down.

Likewise, on another occasion, I was at a seminar and the presenter failed to give his demo to the audience. The reason was that there had been a planned power shutdown at his office, of which he had been unaware. Shut downs should be minimized and in some cases eliminated. Where shutdowns are necessary for upgrades and regular maintenance, alternative systems should be operational during the shutdown phase.

A rule of thumb is to achieve an availability of five-nines: the system is available 99.999% of the time. This equates to a stringent downtime of 6 seconds in a week or 5 minutes 15 seconds in a year. System availability is dependent on the availability of its components. A chain is only as strong as its weakest link. Thus, if a system needs five-nines availability, then the software should provide six-nines availability and the hardware should provide six-nines availability (0.999999 x 0.999999 = 0.999998).

This implies that the system does not breakdown, behave erratically or respond in unexpected ways. The system should perform its job as expected. It is functionally correct and of a certain guaranteed quality. Availability and reliability when taken together make for robustness of the system.

Some months ago I was writing a piece of software for decoding MAC-hs PDUs. These are PDUs received on the DL on HSDPA transport channel HS-DSCH. One of the requirements from this decoder was that it should be robust. This meant that given an illegal combination of input stream and channel configuration, the decoder does not crash the system. It meant that the decoder identifies the set of possible errors and takes corrective/preventive actions as appropriate. This meant that given any random input (used for testing robustness), the decoder does a best-effort decoding without crashing the thread.

In telephone systems, reliability means that calls are handled as expected – no wrong numbers, no premature call termination, no temporary loss of the link, no crosstalk, no loss of overall quality. A typical goal is achieve four-nines reliability: only one in 10,000 calls is mishandled. However, for financial transactions the requirement is a lot more stringent.

This is important from the outset but this is often overlooked. When an new cellular operator sources for equipment, his requirement may be only a dozen base stations to cover a city. His expected subscriber base may be no more than a million. What happens if his growth is phenomenal? What if within the first year of operation he has to expand his network, improve coverage and capacity, and satisfy more than 10 million subscribers? He would like to scale his current system rather than purchase a new and bigger one.

Scalability is often in reference to architecture. A system that has five units can scale to fifty easily because the architecture allows for it. On the other hand, a system designed specifically for five units cannot scale to fifty because its architecture is inadequate. Changing system architecture in a fundamental during system resizing is not scalability. Scalability implies resizing with minimal effort. The effort is minimal because the architecture takes care of the resizing. There is no extra design or development, only deployment and operation.

Scalability applies downwards as well. For example, a switch may have 100 parallel mobules operating at full capacity. At off-peak times, it should be possible to power down 80 modules and run only on the remaining 20. I refer to this as operational scalability. Overall, scalability is a highly attractive feature for any customer. The architecture may not necessarily be fixed. It could be configured appropriately to suit the purpose at a particular scale.

As an example, I recently came across a system with multiple threads and synchronous messaging from each thread. The system was not scalable for two reasons:

  1. Each user had a dedicated thread, all running on the same CPU. They were all doing the same task, only for different users. This is acceptable for a few users but with thousands of users, the overhead from context-switching will be high. A distributed architecture might also suit.
  2. Synchronous communication meant that the thread blocked while waiting for a reply from an external entity. Such idle times are not a problem for a lightly loaded system. When thousands of users are involved, asynchronous messaging must be preferred to make best use of idle times. What this means is that requests and responses are pipelined. Requests could be on one thread and responses processed in another thread.

One easy way to explain capacity is to look at the PC market. Intel 386 is far below in terms of performance when compared to Intel Pentium 4. However, if we consider the overall system capacity we will realize that the improvement is less than what the numbers suggest. As processors have got better, so too is the demand from applications. Today’s applications are probably running at about the same speed as yesterday’s applications. Only thing that’s really improved is the user experience. So, has the capacity of processors really improved?

In fact, capacity is closely linked to scalability. What if the system can scale if capacity drops due to increased overheads? What if the system can scale if lot more processors are required than what the competition can offer?

This relates to maintenance and upgrade of software. No system today is quite simple. Complex systems must contain intelligent partitions and well-defined interfaces. These are generally managed by large teams. While no single person will grasp the details of the entire system, he will be an expert in his own component or sub-system. Activities happen in parallel – bug fixes, new feature releases, modifying requirements or patches. Good documentation is needed. Good processes leading to better productivity will increase the lifespan of a software system.

Some elements of a good software architecture are:

  • well-defined interfaces
  • layering (vertical separation)
  • partitioning (horizontal separation)
  • high cohesion (within components)
  • loose coupling (between components)

In conclusion, making a carrier-grade system does not happen right at the start. It is a process that involves continuous improvement. A good design is essential. Such a design will eventually meet all the five requirements that we have discussed. There are necessary choices to be made along the way. For example, a layered architecture suits well for large systems managed by large teams. It has good productivity but some parts of the system could have been of a higher capacity had they been of a leaner design. Likewise, application frameworks seek to provide a uniform API for applications. This increases productivity but at the expense of capacity.

Read Full Post »