Extras-testing/QA Checklist/QA Improvements

Latest revision as of 13:12, 12 May 2010

Note: Work in progress, none of the action points below are definitive.

1 Roles
2 Automatic checks/Autobuilder
3 Tool for easy on device testing
4 Package Interface
5 Check List
6 Testing Squad
- 6.1 Testing Squad mailing list

[edit] Roles

Maintainer - the owner of the package under testing
Tester - Any community member
Master/Admin - selected testing squad members.

[edit] Automatic checks/Autobuilder

~~Bugtracker field~~ - Done
~~That the description field is not empty~~ - Done
Require description field content check only if description has changed
Optified and dependencies are optified too
License files included and headers have copyright/license.

[edit] Tool for easy on device testing

Some easy way to capture a log of power usage, used files and open ports.

[edit] Package Interface

~~Changelog should be displayed~~ - Done
A list of application specific test cases should be displayed (if available. if not available testers should be able to create one.)
If the package is a library there should be shown packages of application using this library. So everybody can test libraries indirect on application level.
~~Votes should be changeable~~ - Done
Each package that enters or leaves testing triggers a e-mail for the testing squad list
Link to Wiki so that details of test criteria are always accessible to new testers

[edit] Thumbs Up

Collapsable check list will appear, testers should mark the fields tested.
Promotion should occur at +10 karma (if there's at least one completed check list ???)
- or maybe it would be better to have one check per testing task expect for "Working provided features." which should have for example 5 votes. This would be better to make sure that there is no functionality blockers than just one cheking the functionality. (see detailed description in http://talk.maemo.org/showpost.php?p=481536&postcount=55 )

[edit] Thumbs Down

Testers must comment on thumbs down.
Maintainers thumb down will demote the application from the testing queue.
X (fix me) thumbs down will demote the app.

[edit] Demotion

Packages can be demoted at any time by their maintainers (Implemented) or by a member of the testing squad (demotions should be advertised in the testing squad list).
When demoting a package there's an option to keep the current app karma (minor issues) or reset it (big blockers), and a text field where should be added the reason for demotion.

[edit] Speed Promotion

Maintainer can request speed promotion through interface in well defined cases: critical bug in "Extras"-version needs urgent fix; only cosmetic changes (new translations, icons, package description,...);
Changes must be easily visible/documented (run diff agains version in Extras)
Speed promotion is done by selected members of the testing sqad. No extra requirements like "10 days" or minimum package karma. ossi1967

[edit] Speed Promotion (alternative)

Alternate suggestion: given that the smallest change by a developer can cause a serious regression, and there's no way round that - is that once a package reaches the "tipping point" (say, 5 days and 8 votes) another version of the package is let into Extras-testing.

However, people can still rate the earlier version (although not install it) and get it through; whilst the newer version starts its QA process. Obviously if there's a bug, the developer can demote their earlier version and prevent it going through to Extras. --Jaffa 13:12, 12 May 2010 (UTC)

[edit] Check List

1. [ ] Licensing ok.
2. [ ] Description field ok.
--
3. [ ] Announced features available.
4. [ ] Working provided features.
--
5. [ ] No performance problems.
6. [ ] No power management issues.
7. [ ] No known security risks.
8. [ ] No illegal/dubious content.

Additional comments:

Put [x] for those tests you have done, elaborate on separate row if the test is FAIL.
Vote up if there were no FAILs. If there was even one FAIL, vote down.
UI usability issues cannot be used as a reason for vote down.
Always test functionality - that is, run the program and try if it works as it should.

imaginary example:

1. [x] Licensing ok.
2. [x] Description field ok.
 FAIL: the package doesn't have a description.
--
3. [ ] Announced features available.
4. [x] Working provided features.
 FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123). 
 FAIL: When exporting file the program crashes (see bug: http://url/456)
--
5. [ ] No performance problems.
6. [ ] No power management issues.
7. [ ] No known security risks.
8. [ ] No illegal/dubious content.

Additional comments: 
I liked the program, even though, as is, I have to vote it down due to the bugs. I added few usability enhancements to bugzilla, see http://url/567 and http://url/678

[edit] Testing Squad

Can demote packages when there are known blockers.
Can promote packages when they are stuck in the testing queue for a while without any known blocker.
Can promote packages in speed promotion process ossi1967

[edit] Testing Squad mailing list

Public mailing list where are discussed any situation/issue concerning the applications in the maemo.org repositories.
Receives an automatic notification for each package that enters testing, is demoted or is promoted.

Retrieved from "http://bugs.maemo.org/Extras-testing/QA_Checklist/QA_Improvements"

Category: Quality Assurance

This page was last modified on 12 May 2010, at 13:12.
This page has been accessed 36,148 times.

@@ Line 1: / Line 1: @@
 '''Note: Work in progress, none of the action points below are definitive.'''
+== Roles ==
+* Maintainer - the owner of the package under testing
+* Tester - Any community member
+* Master/Admin - selected testing squad members.
 == Automatic checks/Autobuilder ==
+* <s>Bugtracker field</s> - '''Done'''
+* <s>That the description field is not empty</s> - '''Done'''
+* Require description field content check only if description has changed
+* Optified and dependencies are optified too
+* License files included and headers have copyright/license.
-* Bugtracker field - TBD
+== Tool for easy on device testing ==
-* Optified - TBD
+* Some easy way to capture a log of power usage, used files and open ports.
-** also check opt. of other packages pulled in as dependencies
 == Package Interface ==
+* <s>Changelog should be displayed</s> - '''Done'''
-* Changelog should be displayed.
+* A list of application specific test cases should be displayed (if available. if not available testers should be able to create one.)
-* Votes should be changeable.
+* If the package is a library there should be shown packages of application using this library. So everybody can test libraries indirect on application level.
+* <s>Votes should be changeable</s> - '''Done'''
 * Each package that enters or leaves testing triggers a e-mail for the testing squad list
 * Link to Wiki so that details of test criteria are always accessible to new testers
 === Thumbs Up ===
 * Collapsable [[#Check_List | check list]] will appear, testers should mark the fields tested.
 * Promotion should occur at +10 karma (if there's at least one completed check list ???)
+** or maybe it would be better to have one check per testing task expect for  "Working provided features." which should have for example 5 votes. This would be better to make sure that there is no functionality blockers than just one cheking the functionality. (see detailed description in http://talk.maemo.org/showpost.php?p=481536&postcount=55 )
 === Thumbs Down ===
 * Testers must comment on thumbs down.
 * Maintainers thumb down will demote the application from the testing queue.
@@ Line 26: / Line 35: @@
 === Demotion ===
+* Packages can be demoted at any time by their maintainers ('''Implemented''') or by a member of the testing squad (demotions should be advertised in the testing squad list).
+* When demoting a package there's an option to keep the current app karma (minor issues) or reset it (big blockers), and a text field where should be added the reason for demotion.
-* Packages can be demoted at any time by his maintainers or by a member of the testing squad (demotions should be advertised in the testing squad list).
+=== Speed Promotion ===
-* When demoting a package there's a option to keep the current app karma (minor issues) or reset it (big blockers), and a text field where should be added the reason for demotion.
+* Maintainer can request speed promotion through interface in well defined cases: critical bug in "Extras"-version needs urgent fix; only cosmetic changes (new translations, icons, package description,...);
+* Changes must be easily visible/documented (run diff agains version in Extras)
+* Speed promotion is done by selected members of the testing sqad. No extra requirements like "10 days" or minimum package karma. [[User:ossi1967|ossi1967]]
-== Check List ==
+=== Speed Promotion (alternative) ===
+Alternate suggestion: given that the smallest change by a developer can cause a serious regression, and there's no way round that - is that once a package reaches the "tipping point" (say, 5 days and 8 votes) another version of the package is let into Extras-testing.
+However, people can still rate the earlier version (although not install it) and get it through; whilst the newer version starts its QA process. Obviously if there's a bug, the developer can demote their earlier version and prevent it going through to Extras. --[[User:jaffa|Jaffa]] 13:12, 12 May 2010 (UTC)
+== Check List ==
 <pre>
 . [ ] Licensing ok.
-. [ ] No illegal/dubious content.
+. [ ] Description field ok.
-. [ ] Working provided features.
+--
-. [ ] No missing announced features.
+. [ ] Announced features available.
-. [ ] Optification ok.
+. [ ] Working provided features.
-. [ ] No performance problems.
+--
-. [ ] No power management issues.
+. [ ] No performance problems.
-. [ ] No known security risks.
+. [ ] No power management issues.
+. [ ] No known security risks.
+. [ ] No illegal/dubious content.
 Additional comments:
@@ Line 49: / Line 68: @@
 * UI usability issues cannot be used as a reason for vote down.
 * Always test functionality - that is, run the program and try if it works as it should.
-* Pls consider adding "Package has description field" as additional item in the check list. Packages without or packages describing themselves as "''packagename'' application" shouldn't go to Extras as this is so easily done.
 imaginary example:
 <pre>
 . [x] Licensing ok.
-. [ ] No illegal/dubious content.
+. [x] Description field ok.
-. [x] Working provided features.
+ FAIL: the package doesn't have a description.
+--
+. [ ] Announced features available.
+. [x] Working provided features.
   FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123).
   FAIL: When exporting file the program crashes (see bug: http://url/456)
-. [ ] No missing announced features.
+--
-. [x] Optification ok.
+. [ ] No performance problems.
- FAIL: the package uses 1512kb from root.
+. [ ] No power management issues.
-. [ ] No performance problems.
+. [ ] No known security risks.
-. [ ] No power management issues.
+. [ ] No illegal/dubious content.
-. [ ] No known security risks.
 Additional comments:
@@ Line 70: / Line 90: @@
 == Testing Squad ==
+* Can demote packages when there are known blockers.
-* Can demote packages when there's blockers.
 * Can promote packages when they are stuck in the testing queue for a while without any known blocker.
+* Can promote packages in speed promotion process [[User:ossi1967|ossi1967]]
 === Testing Squad mailing list ===
+* Public mailing list where are discussed any situation/issue concerning the applications in the maemo.org repositories.
-* Public mailing list where are discussed concerns/improvements to the QA process.
+* Receives an automatic notification for each package that enters testing, is demoted or is promoted.
-* Receives a notification for each packages that enters testing, is demoted or is promoted.
+[[Category:Quality Assurance]]

Personal tools

Navigation

Views