Software Testing: The Basics of the Trade |
|
|
|
|
| Gaia Asher |
| September 2005, Galiel.Net, Paperback, 175 pages, ISBN 0977036405
|
|
|
|
 |
|
In many companies, software testers are simply considered software
developers with a lower pay. However, that’s not the case. While
it is true that software testing involves a lot of programming, it
also requires a significantly different mind set and skills. Let’s
illustrate this with a story:
Dark Ages. First, the developer arrives at the site and starts
to plan the castle. He thinks, "Let’s put it on top of that
hill, it’s the highest and provides a good, defendable position.
It will also border with a river and so the defenders will always
have water. Walls? Five meters thick and enforced concrete should
do the trick. Gates… steel, half a meter thick. Good."
Then the test team arrives on horses in a full armor, and the
test lead starts to look at the castle. His thoughts are a bit different, "Ok,
the position is strong, cannot take it over with a direct attack.
What about the tower? Ladders will not work, too tall… May
be walls? Enforced concrete, five meters thick, cannot use a ram.
No, no weakness here. Gates… hm-m-m… Steel, half a
meter thick, cannot break through. By the way, did they lock them?"
This book is devoted to software testing as a trade, something that
people do for a living. And the primary thing that separates the trade
and the art is that you cannot limit yourself to a single technique,
area, or dimension of your trade. Van Gogh could paint some of his
pictures in blue. It was later called a "blue period" and these pictures
are in the best museums of the world, because it was art. Software
testers have to use all of the spectrum and keep in mind all the dimensions
of software testing, because it’s their trade.
A complete up-to-date picture of the dimensions of software testing
is what this book is about.
|
 |
|
1 PREFACE ____________________________________ 3
2 AT A GLANCE ________________________________ 5
3 TABLE OF CONTENTS ________________________ 9
4 INTRODUCTION _____________________________ 17
5 WHY TEST SOFTWARE? _____________________ 19
5.1 WHY TEST SOFTWARE? ______________________ 19
5.2 LIMITATIONS OF TESTING ____________________ 20
5.2.1 The First Law of Software Programming______ 20
5.2.2 Nothing is ever perfect ____________________ 21
5.2.2.1 Requirements _______________________ 22
5.2.2.2 Design ____________________________ 23
5.2.2.2.1 Requirements context, missed or ignored
requirements ______________________________ 24
5.2.2.2.2 Overusing formal verification tools
without proper qualification __________________ 27
5.2.2.2.3 Late design changes and code fixes ___ 29
5.2.2.3 Code______________________________ 29
5.2.2.4 Platform ___________________________ 30
5.2.2.5 Tests______________________________ 30
5.2.2.6 Users _____________________________ 31
5.2.2.6.1 Monkey_________________________ 32
10
5.2.2.6.2 Farmer__________________________ 33
5.2.2.6.3 Hacker__________________________ 34
5.2.2.7 The Product Team ___________________ 34
5.2.3 Richness of possibilities ___________________ 34
5.2.3.1 Code paths, state transitions____________ 35
5.2.3.2 Inputs (including input editing and timing) 36
5.2.3.3 Data in the environment_______________ 37
5.2.3.4 In essence – you cannot try everything ___ 38
5.3 ALTERNATIVE TO TESTING – PROVING __________ 38
5.4 THREE OBJECTIVES OF TESTING SOFTWARE ______ 39
5.4.1 So, why? _______________________________ 39
5.4.2 Objective #1: Quality Bar__________________ 39
5.4.3 Objective #2: Acceptance __________________ 41
5.4.4 Objective #3: Baseline ____________________ 41
6 WHAT’S SOFTWARE TESTING ABOUT? _______ 43
6.1 THE DIFFERENCE BETWEEN A TESTER AND A
DEVELOPER _____________________________________ 43
6.1.1 A Castle _______________________________ 43
6.1.2 Debugging is not testing___________________ 44
6.1.2.1 Goals _____________________________ 44
6.1.2.2 Actor _____________________________ 45
6.1.2.3 Tools _____________________________ 45
6.1.2.4 Results ____________________________ 45
6.1.3 Super-pessimism – Testing that the glass is half
empty 45
6.2 THE THREE-HEADED DRAGON OF SOFTWARE
DEVELOPMENT ___________________________________ 46
6.2.1 Software developer _______________________ 46
6.2.2 Software architect, analyst, project manager___ 47
6.2.3 Software tester __________________________ 48
6.3 THE LANGUAGE OF SOFTWARE TESTING ________ 48
6.3.1 The bug: what is it? ______________________ 48
6.3.2 Test case _______________________________ 49
6.3.3 Test plan _______________________________ 50
6.3.4 Bug tracking ____________________________ 50
6.3.5 Version control __________________________ 51
6.3.6 Test automation _________________________ 53
6.4 A BUG’S LIFE _____________________________ 54
11
6.4.1 Bug opened _____________________________ 54
6.4.2 Bug approved ___________________________ 55
6.4.3 Bug fixed_______________________________ 55
6.4.4 Bug verified and closed ___________________ 56
6.5 SEVERITY OF BUGS _________________________ 56
6.5.1 Know Thine User!________________________ 57
6.6 THE KINDS OF BUGS ________________________ 59
6.6.1 Code defect _____________________________ 59
6.6.2 Crash _________________________________ 60
6.6.3 A special kind: Regression _________________ 61
6.6.4 Security ________________________________ 61
6.6.5 Performance-related______________________ 62
6.6.5.1 Throughput (performance)_____________ 62
6.6.5.2 Stress _____________________________ 63
6.6.5.3 Volume____________________________ 64
6.6.5.4 Scalability _________________________ 64
6.6.5.5 Resources consumption (leaks) _________ 65
6.6.6 Usability _______________________________ 67
6.6.7 Documentation __________________________ 69
6.6.8 Test “issue” ____________________________ 70
6.6.8.1 Bugs in tests and frameworks __________ 70
6.6.8.2 Mixing up a bug and a feature __________ 70
7 STAGES OF SOFTWARE DEVELOPMENT FOR A
TESTER _________________________________________ 72
7.1 CHALLENGING REQUIREMENTS________________ 72
7.2 CHALLENGING SPECIFICATIONS _______________ 73
7.2.1 Challenging assumptions __________________ 74
7.2.2 Challenging completeness _________________ 75
7.2.3 Challenging behavior _____________________ 77
7.2.4 Challenging usability _____________________ 78
7.2.5 Challenging testability ____________________ 78
7.2.6 That’s not the end ________________________ 79
7.3 CHALLENGING ARCHITECTURE AND TECHNICAL
DESIGN 79
7.4 TESTING CODE_____________________________ 81
7.5 CHALLENGING DOCUMENTATION ______________ 81
7.6 RELEASE _________________________________ 83
8 DIMENSIONS OF SOFTWARE TESTING _______ 85
12
8.1 WHY “DIMENSIONS”? _______________________ 85
8.2 BY OBJECT _______________________________ 86
8.2.1 Positive testing – capabilities _______________ 86
8.2.2 Negative testing – environment _____________ 86
8.3 BY SOURCE CODE VISIBILITY_________________ 86
8.3.1 Black (“opaque”) box testing_______________ 86
8.3.2 White (glass, clear) box testing _____________ 87
8.3.3 “Gray” (“Translucent”) box testing _________
88
8.4 BY LEVEL ________________________________ 88
8.4.1 Unit testing _____________________________ 88
8.4.2 Component testing _______________________ 89
8.4.3 Integration testing________________________ 90
8.4.4 System testing ___________________________ 91
8.4.4.1 Scenario-based ______________________ 92
8.4.4.2 Free-form __________________________ 92
8.5 BY PLACE ________________________________ 93
8.5.1 Product Team Testing (vendor) _____________ 93
8.5.2 Customer (user) Side Testing _______________ 93
8.5.3 Third Party Testing (certification) ___________ 94
8.6 BY PURPOSE ______________________________ 95
8.6.1 Functional testing________________________ 95
8.6.2 Regression testing________________________ 96
8.6.3 Performance related testing ________________ 97
8.6.3.1 Performance (throughput) testing _______ 97
8.6.3.2 Volume (size) testing _________________ 97
8.6.3.3 Load (stress) testing __________________ 98
8.6.3.4 Scalability testing____________________ 98
8.6.3.5 Resource consumption (including leaks)
testing 98
8.6.3.6 Sizing guidance _____________________ 99
8.6.4 Conformance testing______________________ 99
8.6.4.1 Standards _________________________ 100
8.6.4.2 Legislation ________________________ 100
8.6.4.3 Coding policy testing ________________ 100
8.6.5 Security testing _________________________ 100
8.6.5.1 Functional security testing ____________ 100
8.6.5.2 Security model and STRIDE __________ 101
8.6.5.3 Security penetration testing ___________ 103
8.6.6 Globalization testing_____________________ 103
13
8.6.6.1 Encoding _________________________ 103
8.6.6.2 Watch your language ________________ 104
8.6.6.3 Formats __________________________ 106
8.6.6.4 More calendar fun __________________ 106
8.6.6.5 Units_____________________________ 107
8.6.6.6 Strings ___________________________ 108
8.6.6.7 Length of text______________________ 108
8.6.6.8 Making of an error message___________ 109
8.6.6.9 That’s not all ______________________ 110
8.6.7 Localization testing______________________ 110
8.6.7.1 Cultural differences _________________ 111
8.6.8 Platform and dependencies testing__________ 112
8.6.9 Pseudo-testing _________________________ 114
8.6.9.1 Usability__________________________ 114
8.6.9.2 Reliability_________________________ 115
8.6.9.3 Robustness ________________________ 115
8.7 BY ATTACK SURFACE______________________ 116
8.7.1 What is an attack surface _________________ 116
8.7.2 Input _________________________________ 117
8.7.2.1 Input – GUI _______________________ 117
8.7.2.2 Input – API________________________ 118
8.7.2.3 Input – Files and DB content __________ 119
8.7.3 Environment ___________________________ 119
8.7.3.1 Environment – Resources (storage, memory,
etc.) 119
8.7.3.2 Environment – Access rights __________ 120
8.7.3.3 Environment – Network______________ 120
8.7.3.4 Environment – Files and folders _______ 120
8.7.3.5 Environment – Media _______________ 121
8.7.3.6 Environment – Output devices_________ 121
8.7.3.7 Environment – Platform______________ 122
8.7.4 User _________________________________ 122
8.7.4.1 User – monkey _____________________ 122
8.7.4.2 User – farmer ______________________ 123
8.7.4.3 User – hacker ______________________ 123
8.8 BY FAULT TYPE __________________________ 123
8.8.1 A word on the fault model_________________ 123
8.8.2 Boundary conditions_____________________ 124
8.8.3 Arithmetic overflows_____________________ 125
14
8.8.4 Buffer overflows ________________________ 126
8.8.5 Race conditions_________________________ 127
8.8.6 Uninitialized variables ___________________ 129
8.8.7 Uncaught exceptions_____________________ 130
8.8.8 Resource allocation _____________________ 130
8.8.9 That’s not all again______________________ 130
9 QUALITY OF TESTING ______________________ 131
9.1 HUH? ___________________________________ 131
9.2 CODE COVERAGE__________________________ 131
9.3 SOFTWARE METRICS _______________________ 132
9.4 TEST PLAN REVIEW ________________________ 133
9.5 SEEDING BUGS AND CODE MUTATION__________ 134
9.6 TESTABILITY _____________________________ 134
9.7 BETA RELEASE/BETA TESTING _______________ 136
10 POST-RELEASE FUN ________________________ 138
10.1 DEPLOYMENT TESTING _____________________ 139
10.2 COMPATIBILITY TESTING ___________________ 139
10.3 SYSTEM TESTING__________________________ 139
10.4 STAGING AND PRODUCTION ENVIRONMENT _____ 140
11 TESTING SOFTWARE IN A GLOBAL TEAM ___ 141
11.1 TECHNICALLY SPEAKING ___________________ 141
11.2 HUMAN-ORIENTED ________________________ 142
11.2.1 Time zone differences. _________________ 142
11.2.2 Work relations and cultural differences____ 142
11.2.3 Life in a global org____________________ 143
12 SURVIVING AS A TESTER ___________________ 145
12.1 FUZZY VALUE IN THE SOFTWARE DEVELOPMENT_ 145
12.2 COST/VALUE BALANCE _____________________ 146
12.3 RELIGIONS AND CONFESSIONS _______________ 147
12.4 IEEE ROLE ______________________________ 151
12.5 OFFICE POLITICS __________________________ 151
12.6 OUTSOURCING AND JOB SECURITY ____________ 154
12.7 SEGREGATION ____________________________ 157
12.8 FUTURE OF THE TRADE _____________________ 158
12.8.1 From nobility to peasants_______________ 158
15
12.8.2 Crushing craftsmen ___________________ 161
12.8.3 Assault on software development _________ 162
12.8.4 What’s next?_________________________ 165
13 APPENDIX A. TEST CASE____________________ 168
14 APPENDIX B. BIBLIOGRAPHY _______________ 170
14.1 STANDARDS AND CLASSICS__________________ 170
14.2 EXTREME TESTING & RELATED ______________ 171
14.3 OTHER__________________________________ 172
15 ENDNOTES _________________________________ 174 |
|
 |
|
| From the author:
Got an interview for a software tester or software design
engineer in test (SDET) position? Read this book and go
for the kill.
Got a phone interview for a software tester or SDET position? Print
the table of contents available on the publisher's site for free, and
use it as a plan and reference. You'll be back ordering soon.
Got a software tester position? Read this book to
keep it.
Got a software project to manage with the testing outsourced
to India? Read this book to know whether they do it right
or screw you and themselves over.
Have a test team in India to sell their services in the United
States? Bulk orders are welcome.
Still in college? Do you take a course on software
design and development? Do you have a test for it?
New to the software development? Fresh from college? Read
this book to learn what you are missing (unless you read it in the
college).
Have 20 years of experience in the software industry as I
do? Know it all and observed software testing growing from Glen Myer's "The
Art of Software Testing" to its modern state? Read this
book anyway. You'll enjoy it a lot. And you'll be able to fully appreciate
it.
Am I bragging? Yes. You would be too, if you wrote
such a good book.
Who is this book for? Software testers, developers,
Computer Science and Information Technology students, software project
managers.
|
|
