Extras-testing/QA Checklist/QA Improvements

Note: Work in progress, none of the action points below are definitive.

Contents

Roles

  • Maintainer - the owner of the package under testing
  • Tester - Any community member
  • Master - selected persons, for example with high testing karma or testing squad members.

Automatic checks/Autobuilder

  • Bugtracker field
  • Optified and dependencies too
  • License files included and headers have copyright/license.

Tool for easy on device testing

  • Some easy way to capture a log of power usage, used files and open ports.

Package Interface

  • Changelog should be displayed.
  • Votes should be changeable.
  • Each package that enters or leaves testing triggers a e-mail for the testing squad list
  • Link to Wiki so that details of test criteria are always accessible to new testers

Thumbs Up

  • Collapsable check list will appear, testers should mark the fields tested.
  • Promotion should occur at +10 karma (if there's at least one completed check list ???) .. or for every checklist item, one check is enough expect for number 3. "Working provided features." which should have for example 5 or ten votes. This would be better to make sure that there is no functionality blockers.

Thumbs Down

  • Testers must comment on thumbs down.
  • Maintainers thumb down will demote the application from the testing queue.
  • X (fix me) thumbs down will demote the app.

Demotion

  • Packages can be demoted at any time by his maintainers or by a member of the testing squad (demotions should be advertised in the testing squad list).
  • When demoting a package there's a option to keep the current app karma (minor issues) or reset it (big blockers), and a text field where should be added the reason for demotion.

Check List 

1. [ ] Optification ok.
2. [ ] Licensing ok.
--
3. [ ] Announced features available.
4. [ ] Working provided features.
--
5. [ ] No performance problems.
6. [ ] No power management issues.
7. [ ] No known security risks.
8. [ ] No illegal/dubious content.

Additional comments:
  • Put [x] for those tests you have done, elaborate on separate row if the test is FAIL.
  • Vote up if there were no FAILs. If there was even one FAIL, vote down.
  • UI usability issues cannot be used as a reason for vote down.
  • Always test functionality - that is, run the program and try if it works as it should.
  • Pls consider adding "Package has description field" as additional item in the check list. Packages without or packages describing themselves as "packagename application" shouldn't go to Extras as this is so easily done.

imaginary example: 

1. [x] Licensing ok.
2. [x] Optification ok.
 FAIL: the package uses 1512kb from root.
--
3. [ ] Announced features available.
4. [x] Working provided features.
 FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123). 
 FAIL: When exporting file the program crashes (see bug: http://url/456)
--
5. [ ] No performance problems.
6. [ ] No power management issues.
7. [ ] No known security risks.
8. [ ] No illegal/dubious content.

Additional comments: 
I liked the program, even though, as is, I have to vote it down due to the bugs. I added few usability enhancements to bugzilla, see http://url/567 and http://url/678

Testing Squad

  • Can demote packages when there's blockers.
  • Can promote packages when they are stuck in the testing queue for a while without any known blocker.

Testing Squad mailing list

  • Public mailing list where are discussed concerns/improvements to the QA process.
  • Receives a notification for each packages that enters testing, is demoted or is promoted.
Retrieved from "http://bugs.maemo.org/Extras-testing/QA_Checklist/QA_Improvements"