Extras-testing/QA Checklist/QA Improvements
Note: Work in progress, none of the action points below are definitive.
Contents |
Roles
- Maintainer - the owner of the package under testing
- Tester - Any community member
- Master/Admin - selected testing squad members.
Automatic checks/Autobuilder
- Bugtracker field
- Optified and dependencies too
- License files included and headers have copyright/license.
Tool for easy on device testing
- Some easy way to capture a log of power usage, used files and open ports.
Package Interface
- Changelog should be displayed.
- A list of application specific test cases should be displayed (if available. if not available testers should be able to create one.)
- If the package is a library there should be shown packages of application using this library. So everybody can test libraries indirect on application level.
- Votes should be changeable.
- Each package that enters or leaves testing triggers a e-mail for the testing squad list
- Link to Wiki so that details of test criteria are always accessible to new testers
Thumbs Up
- Collapsable check list will appear, testers should mark the fields tested.
- Promotion should occur at +10 karma (if there's at least one completed check list ???)
- or maybe it would be better to have one check per testing task expect for "Working provided features." which should have for example 5 votes. This would be better to make sure that there is no functionality blockers than just one cheking the functionality. (see detailed description in http://talk.maemo.org/showpost.php?p=481536&postcount=55 )
Thumbs Down
- Testers must comment on thumbs down.
- Maintainers thumb down will demote the application from the testing queue.
- X (fix me) thumbs down will demote the app.
Demotion
- Packages can be demoted at any time by his maintainers or by a member of the testing squad (demotions should be advertised in the testing squad list).
- When demoting a package there's a option to keep the current app karma (minor issues) or reset it (big blockers), and a text field where should be added the reason for demotion.
Check List
1. [ ] Licensing ok. 2. [ ] Description field ok. -- 3. [ ] Announced features available. 4. [ ] Working provided features. -- 5. [ ] No performance problems. 6. [ ] No power management issues. 7. [ ] No known security risks. 8. [ ] No illegal/dubious content. Additional comments:
- Put [x] for those tests you have done, elaborate on separate row if the test is FAIL.
- Vote up if there were no FAILs. If there was even one FAIL, vote down.
- UI usability issues cannot be used as a reason for vote down.
- Always test functionality - that is, run the program and try if it works as it should.
imaginary example:
1. [x] Licensing ok. 2. [x] Description field ok. FAIL: the package doesn't have a description. -- 3. [ ] Announced features available. 4. [x] Working provided features. FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123). FAIL: When exporting file the program crashes (see bug: http://url/456) -- 5. [ ] No performance problems. 6. [ ] No power management issues. 7. [ ] No known security risks. 8. [ ] No illegal/dubious content. Additional comments: I liked the program, even though, as is, I have to vote it down due to the bugs. I added few usability enhancements to bugzilla, see http://url/567 and http://url/678
Testing Squad
- Can demote packages when there is known blockers.
- Can promote packages when they are stuck in the testing queue for a while without any known blocker.
Testing Squad mailing list
- Public mailing list where are discussed any situation/issue concerning the applications in the maemo.org repositories.
- Receives a notification for each packages that enters testing, is demoted or is promoted.