Smoke testing of all base utilities
Student: Shivansh Rai (<shivansh AT freebsd DOT org>)
- Mentors:
Alan Somers (<asomers AT freebsd DOT org>)
Brooks Davis (<brooks AT freebsd DOT org>)
Ngie Cooper (<ngie AT freebsd DOT org>)
Contents
Overview
Smoke testing is a set of light tests which are done for checking basic functionalities of a software to ascertain if the crucial functions work correctly.
FreeBSD currently has a set of tests which are run using the kyua framework. These tests need to be first installed individually before they can be used for testing. This proves to be problematic in cases when direct testing of some newly installed or updated utility has to be performed. It makes testing changes to utilities and libraries difficult as one would like to perform tests (e.g. to ensure a proper build environment) before proceeding for installation.
Project Description
This project aims to develop a test infrastructure and automation tool along with basic tests to verify if all the base utilities in FreeBSD are linked properly. The testing framework is supposed to ease the process of writing "trivial" test cases which will be run in a completely automated and developer friendly manner without need for any prior installation. Once integrated, the tool will also facilitate further development of tests.
Implementation Details
The following steps will be performed in order to understand the problem at hand and the approaches which can be used -
- Note how testing has been traditionally done in FreeBSD.
Study the kyua framework and the approaches currently being used for testing.
- Study the current problems faced as a developer or tester when writing new test cases, and the problems faced as a release engineer or tester when executing existing test cases.
Initially Proposed Test Plan
The file functional_test.c is a simple test file which checks whether the ls utility is properly linked by running trivial commands. The options supported by ls are stored in short_options[] and long_options[] present in functional_test.h.
It should be noted that finally the tests will be automated with appropriately passed options. The above commands will not have to be run for testing individual programs. The test file uses getopt() and getopt_long() for testing the validity of the passed options. If a valid option is passed, the command <utility> --<option(i)> is executed. In case the command fails to execute for a valid option, this will imply that the utility under test is not properly linked.
Populating short_options[] and long_options[]
short_options[] and long_options[] need to be initially populated with a few supported options for all the base utilities. This can be done by using either one or both of the following available approaches -
- Parse man pages for each utility to get the supported options.
- Pass an unsupported option to the utility. This might generate a usage message which can then be parsed. The unsupported option will be chosen experimentally.
An automation script will be written which will populate short_options[] and long_options[] by following the above mentioned approaches and will generate relevant test files. The initial version of the generated test scripts will only test trivial functionalities.
Although the test plan does not check the entire state of application under test, it suffices for the initial purposes of smoke testing.
Revised test plan
The test plan for the project has been completely revamped (a brief overview available here). Comments and suggestions are most welcome!
Expected Results
- An automated tooling to generate test scripts which confirm minimal set of functionalities of a utility to check whether it is properly linked.
- Set of test programs generated by the tool to verify basic functionality of all the base utilities in FreeBSD.
Milestones
Week |
Duration |
Description |
Status |
||
Start of Coding |
|
||||
1 |
May 30 |
Start of coding. |
|
||
Add atf-sh tests |
|
||||
1 |
May 30 - June 5 |
Start adding atf-sh tests to utilities without test coverage. |
|
||
2 |
June 6 - June 12 |
||||
3 |
June 13 - June 19 |
Update smoke testing tool. |
|
||
Fix a bug in the implementation of ln(1). |
|||||
4 |
June 20 - June 26 |
Add functionality to generate trivial tests. |
|
||
First evaluations |
|
||||
Start generating testcases to demonstrate tool behavior in different cases. |
|
||||
5 |
June 27 - July 3 |
Demonstrate negative testcases for ln(1). Updates |
|
||
Support for utilities with no supported options. |
|||||
Collation of common usage message. |
|||||
6 |
July 4 - July 10 |
Demonstrate the tests which the tool can generate using date(1), ln(1) and stdbuf(1). Updates |
|
||
7 |
June 11 - July 17 |
Add functionality for reading annotations. Updates |
|
||
Second evaluations |
|
||||
Bug fixes and code cleanup |
|
||||
8 |
June 18 - July 24 |
Add functionality of dynamically generating annotations. Updates |
|
||
9 |
June 25 - Aug 1 |
Miscellaneous fixes. Updates |
|
||
10 |
Aug 2 - Aug 8 |
Updates to scripts. Updates |
|
||
11 |
Aug 9 - Aug 15 |
||||
12 |
Aug 16 - Aug 22 |
Add functionality to stop test generation during runtime. Updates |
|
||
13 |
Aug 23 - Aug 29 |
||||
Update documentation |
|
||||
Final evaluations |
|
||||
Table design adapted from NonBSMtoBSMConversionTools#Milestones
Deliverables
Deliverables for first evaluation
- Add atf-sh test programs for a subset of base utilities which are currently without any test coverage. The maximum number of utilities possible in the duration will be covered.
- Prepare a basic design of the automation tool. Cover generation of a limited number of test scripts.
Deliverables for second evaluation
- Add functionality to enable users in defining which testcases to ignore while generating tests.
Generate tests for all section 1 utilities and try to identify cases where the tool behaves unexpectedly. Fix such cases.
Deliverables for third evaluation
- Finish adding functionality to halt test generation during runtime.
- Clean up the code and prepare it for merge into the src tree.
Weekly Reports
Note: While visiting links in the following reports to https://shivrai.github.io, they should be renamed to https://shivansh.github.io.
Week |
Report |
1 |
|
2 |
|
3 |
|
4 |
|
5 |
|
6 |
|
7 |
|
8 |
|
The test generation tool was ready at this point |
|
12 |
|
Bugs reported
The Code
Contains the implementation of test generation tool. Also contains the implementation of the initially-proposed test plan.
Differentials Submitted
Week |
Differential |
Status |
1 |
||
2 |
||
3 |
||
4 |
Open |
|
5 |
||
10 |
||
The differential submitted on week 10 lays the basic structure of the generated tests |
||
12 |
Open |
|