A new version of the operating system is being planned for installation into your department’s production…

A new version of the operating system
is being planned for installation into your department’s production
environment. What sort of testing would you recommend is done
before your department goes live with the new version? Identify
each type of testing and describe what is tested. Explain the
rationale for performing each type of testing.
[ your answer goes here
]
Would the amount of testing and types
of testing to be done be different if you were installing a
security patch instead of a new operating system
version? Explain in detail your reasons?
[ your answer goes here
]
Would the amount of testing and types
of testing to be done be different if you were installing an
application patch instead of a new operating
system version? Explain in detail your reasons?
[ your answer goes here
]
What sort of
preparations would you make for an operating
system upgrade on your home PC? Provide some detail.
[ your answer goes here
]

Introduction In the previous sections
we discussed various operational issues. With many of these
operational activities the issue of testing and verification comes
into play. Whenever processes or procedures are changed or added to
a system it is important to verify they work as planned. Of even
more importance is to ensure the new functionality does not cause
any regression in how the system operates. It can be very costly
for a client to install a new system that does not work as planned.
If a client needs to go off line to rollback changes that is one
cost. Having upset customers is another cost. For the manufacturer
that shipped the new system that does not work as planned there is
the cost to re-engineer it to get it right. Also, the credibility
of a manufacturer that ships a poorly tested system can be damaged;
sometimes irreparably if the new system resulted in damage that
could not be reversed, such as if the untested system resulted in a
damaging security breach of data. The idea or excuse that companies
don’t have the time to test because of schedule demands is
ridiculous in my opinion. It demonstrates that leadership does not
understand how to develop systems and the important role testing
plays in the “System Development Life Cycle (SDLC)”.
Description Testing is the
investigation of functionality or behavior or a system to determine
if the system behaves in a predictable manner. The functionality of
a system is measured by comparing the current behavior or outputs
of the system under test to known good behaviors or outputs. Good
or correct system behaviors can be expressed with documentation,
stored benchmarks or other “oracles” or repositories of correct
results that can be used to judge if the system being tested is
producing correct results. There are numerous levels of testing
that can be performed. Each level of testing focuses on a
particular aspect of a systems behavior. Testing should be
performed at various points in the development, deployment or
running of a system. Many organizations will deploy incremental
testing methodologies to ensure that any change in system behavior
is detected as early in the development, deployment or operational
process as possible. There are numerous testing methodologies and
automated testing tools that support the testing process. Test
engineering is a field onto itself that could comprise a semester
topic. What I want to focus on in this section is a sampling of
testing concepts and how they relate to security. An important
point to remember when building a system is that the earlier a
problem is found the cheaper it is to fix. Or stated differently,
the longer a problem exists in the system the more expensive it is
to fix. The following table is based on a NIST study. It provides a
cost attached to finding a problem at various times in the SDLC.
For example, if a problem is found in the requirements and design
phase it has a cost of 1X the cost to repair it. If however, a
problem is found after the system has been released; the cost of
repairing the problem can be extremely high. Having clearly defined
requirements and a comprehensive testing strategy is the most cost
effective way to build a system. Software Development Stage Cost
Requirements and Design 1X Code and Unit Testing 5X Integration and
System Testing 10X Beta Testing 15X Post Release 30X-100X Testing
efforts are directly related to the number of security
vulnerabilities found in a system. The more comprehensive the
testing that is performed the more chance of finding security
vulnerabilities early in the SDLC. There are numerous estimates for
the cost of a security incident. There is much empirical data
showing the cost of a security incident can quickly get into the
millions of dollars. Preventing a security incident means finding
security vulnerabilities. Finding security vulnerabilities early in
the SDLC is desirable. The earlier the vulnerability is found in
the SDLC the cheaper it is to fix. Once a system is released the
cost of a security vulnerability being discovered in the field
quickly mounts. If the vulnerability is exploited into an incident,
the cost further increases.
Relationship of Testing to Quality
and Security In order to ensure a product is of high quality it
must perform as documented and advertised. By utilizing various
testing methodologies the behavior of a product can be verified.
The importance of testing to security is paramount. The tenets of
security are expressed through the attributes of the CIA
(Confidentiality, Integrity, Availability) triad. If any of these
attributes are not adhered to the security of the system is
threatened. Ensuring the tenets of the CIA triad remain intact is
done by testing. The tenets of the CIA triad can be threatened in
the following two ways: _ New functionality is released resulting
in new vulnerabilities. _ New vulnerabilities are discovered and
exploited in existing functionality. Whenever, new functionality is
deployed security testing needs to take place to ensure that all
attributes of the CIA triad are maintained. New functionality can
be introduced into a system by: _ new features implemented in
hardware, software, firmware; _ patches implemented in the system _
configuration changes to the system
Testing Methods There are various
types of testing methods. The waterfall model for developing
systems and testing is a traditional incremental model. In some
cases iterative models of development and testing are more
appropriate. Or in some cases formal testing models utilizing
proofs are desirable. The clarity of the requirements, skill and
proximity of the development team and need for prototype systems
are a few of the criteria that can dictate using one methodology
over another. As part of each programming methodology there are
various techniques used to implement the larger methodology.
Following are three general types of testing techniques that can be
applied to the overall testing strategy.
Black Box Black box
testing tests the functionality of a system. It requires no
understanding of the internal working or implementation of the
system. The tester provides various valid and invalid inputs to the
system and based on the documentation they examine the outputs
determining if the system is performing correctly.
White Box White box
testing requires some understanding of the internal workings and
algorithms of the system. Inputs for the test cases are selected
based on this internal knowledge and therefore specific paths can
be tested. White box testing is a technique that is used in
penetration testing which is discussed below.
Gray Box Gray box
testing involves some understanding of the internal workings of the
system. For example, without knowing detailed algorithm
implementation you might have some understanding of the components
that are used, the configuration of these components and the
default assumptions made in some of the existing black box style
tests. While detailed knowledge is not known there are clearly some
assumptions that could be made in the testing effort to find
possible vulnerabilities in the system.
Levels of Testing
There are numerous levels of testing that can be used throughout
the SDLC (System Development Life Cycle). A testing level can be
thought of as the focus of the testing effort. Following are a few
examples of the levels of testing. There are several more to
consider.
Functionality
Testing This is performed to ensure that functionality
works according to accepted expectations. The expectations of
correct behavior could be based on: design documentation, user
documentation or other published material.
Regression Testing
This is performed to ensure that there is no negative change or
regression in behavior of systems functionality when: new
functionality is deployed, patches are applied or configuration
changes are made to the environment. Regression testing is used for
ensuring there is no regression in: functionality, performance,
system load, security, usability or any other attribute that one
wants to test for.
Performance Testing
This is performed to test how the system performs under various
conditions. The conditions could be based on industry benchmarks,
peak loading, average loading or a variety of other conditions.
Usability Testing
This is performed to test how the system performs from a usability
perspective. Often times a system may perform as designed and
documented but it is not usable from the perspective of the end
user. Perhaps the required commands are awkward to enter, or the
menus and screens difficult to understand. This is sometimes
referred to as testing a system to determine if it is “user
friendly”.
Test Targets Test
targets refer to the part of the system that is being tested. Note
that each test target could be subjected to a particular level of
testing. For example, unit testing could be executed to test for
functionality, regression or performance. These are just a few of
the examples of test targets.
Unit Test This is
performed in the implementation or development phase of a system.
As software routines, hardware or firmware components are being
developed they are incrementally tested. Once they are ready for
integration into the larger system they are said to be ready for
unit test. Unit testing may involve the interfacing and integration
of various routines into a program, run time library or some larger
construct. Unit testing would test for items such as: a routine
accepting the correct number of parameters, are the data types
correct; are the outputs of the routine consistent with inputs.
Integration Test
This is performed when various parts of a system are ready to come
together or interface with one another. The components being
integrated are larger functional units compared to the routines or
procedures that are tested during unit test.
System Test System
test is the testing of a completely integrated system. Ideally, the
system has been incrementally built up from unit test to
integration test with numerous rounds of tests focusing on the
various levels of; functionality, regression, performance and
usability.
Security Test The
importance of security has resulted in a separate branch of testing
called security testing. There are various process and procedures
that are employed in a comprehensive security testing strategy. It
is important to note that security testing is an ongoing process
that needs to occur throughout the life cycle of a system. In other
words, when a system is developed and deployed there should be
numerous iterations of security testing to ensure and validate that
the CIA of the data and the system are intact. Once the system is
deployed and in production it is important that security testing
continues. It is not enough to declare a system has been tested and
certified prior to deployment. The testing effort must be ongoing,
since attackers will be diligent in searching for new
vulnerabilities that were not discovered during development of the
system. Some of the security testing activities that must be done
in an ongoing fashion are: vulnerability assessment and penetration
testing. Vulnerability Assessment Vulnerability assessment is
determining how vulnerable your system is to attack. There are
numerous tools that are used to scan your system to determine the
level of vulnerability. These tools can scan your system for items
such as: open ports, running services, dormant accounts, high
traffic applications and other criteria. Some vulnerability
assessment tools will scan log files looking for certain
activities. These tools then produce reports that can be reviewed
to determine if the vulnerabilities need to be dealt with. There
are also real time vulnerability assessment tools that monitor a
running system for indications of vulnerabilities as well as
incidents. These tools can be programmed to provide real time
alerts in the form of: text messages, mail messages, phone calls,
and process or account shutdowns when vulnerabilities as well as
incidents are detected. Many different types of tools can be used
to detect vulnerabilities in systems. As security related tools
evolve there continues to be a blurring of functionality between
tools such as vulnerability scanners (e.g. Nessus, Nmap), malware
scanners (e.g., Symantec, Avast), intelligent firewalls and IDS
(e.g. Snort) systems.
Penetration Testing
Penetration testing or “pen testing” is a type of testing that
simulates what an attacker would do to break into your system. The
attack being simulated can be an outside attack or an inside
attack. Assumptions made for the inside attack may be that the
attacker possesses detailed knowledge of the system and environment
being attacked. They may have access to user documentation or
design material that an outsider attacker may not have. An insider
may also have more knowledge of the configuration of the systems
under attack that an outsider would not have. Both the insider and
outsider can make use of any tool they want. Network protocol
analyzers such as Wireshark could be used to network traffic and
packets. Vulnerability scanners such as Nessus or Nmap could be
used, particularly by an insider to probe for vulnerabilities on
various systems. Dumpsters could be searched by an outsider for
acquiring important information that missed the shredder, such as a
phone directory, or an insider might have ready access to this
information which would support social engineering attacks such as
phishing to gain passwords or other forms of access. All
information is fair game to simulate an attack. Anything that the
simulator(s) of the attack can obtain will support keeping the
attack real.
Penetration testing is an example of
testing that needs to be done in an ongoing, active and real time
fashion. All of the vulnerabilities in a system are seldom known.
As defects in hardware, software or operational weaknesses are
found new vulnerabilities are also found. These vulnerabilities
need to found by the “good guys” before an attacker finds them and
exploits the vulnerability resulting in a security incident. It is
important that penetration testing is done in an ongoing fashion.
New vulnerabilities are found as system configurations change,
employees change and processes and procedures evolve. The attack
surface is constantly changing and growing. With the changing
attack surface the attack vectors are growing. It is important to
actively monitor your systems for vulnerabilities. Note: You should
never run a penetration test on a system without the permission of
the owner or responsible party for the system. A penetration test
being run without proper permissions is considered a security
breach and can have undesirable consequences for the system being
tested since vulnerabilities will be exposed and valuable data
placed at risk. There can also be undesirable consequences for the
person running the penetration test that does not have the proper
permissions.
Summary In this section we discussed
some aspects of testing and how testing relates to information
security. We tied together the importance of testing and how it
relates to the tenets of CIA. Testing is a topic that could
take