Extras-testing/QA Checklist/QA Improvements
(→Speed Promotion: Alternative) |
|||
| (18 intermediate revisions not shown) | |||
| Line 4: | Line 4: | ||
* Maintainer - the owner of the package under testing | * Maintainer - the owner of the package under testing | ||
* Tester - Any community member | * Tester - Any community member | ||
| - | * Master - selected | + | * Master/Admin - selected testing squad members. |
== Automatic checks/Autobuilder == | == Automatic checks/Autobuilder == | ||
| - | * Bugtracker field | + | * <s>Bugtracker field</s> - '''Done''' |
| - | * Optified and dependencies too | + | * <s>That the description field is not empty</s> - '''Done''' |
| + | * Require description field content check only if description has changed | ||
| + | * Optified and dependencies are optified too | ||
* License files included and headers have copyright/license. | * License files included and headers have copyright/license. | ||
| Line 15: | Line 17: | ||
== Package Interface == | == Package Interface == | ||
| - | * Changelog should be displayed. | + | * <s>Changelog should be displayed</s> - '''Done''' |
| - | * Votes should be changeable | + | * A list of application specific test cases should be displayed (if available. if not available testers should be able to create one.) |
| + | * If the package is a library there should be shown packages of application using this library. So everybody can test libraries indirect on application level. | ||
| + | * <s>Votes should be changeable</s> - '''Done''' | ||
* Each package that enters or leaves testing triggers a e-mail for the testing squad list | * Each package that enters or leaves testing triggers a e-mail for the testing squad list | ||
* Link to Wiki so that details of test criteria are always accessible to new testers | * Link to Wiki so that details of test criteria are always accessible to new testers | ||
| Line 22: | Line 26: | ||
=== Thumbs Up === | === Thumbs Up === | ||
* Collapsable [[#Check_List | check list]] will appear, testers should mark the fields tested. | * Collapsable [[#Check_List | check list]] will appear, testers should mark the fields tested. | ||
| - | * Promotion should occur at +10 karma (if there's at least one completed check list ???) | + | * Promotion should occur at +10 karma (if there's at least one completed check list ???) |
| + | ** or maybe it would be better to have one check per testing task expect for "Working provided features." which should have for example 5 votes. This would be better to make sure that there is no functionality blockers than just one cheking the functionality. (see detailed description in http://talk.maemo.org/showpost.php?p=481536&postcount=55 ) | ||
=== Thumbs Down === | === Thumbs Down === | ||
| Line 30: | Line 35: | ||
=== Demotion === | === Demotion === | ||
| - | * Packages can be demoted at any time by | + | * Packages can be demoted at any time by their maintainers ('''Implemented''') or by a member of the testing squad (demotions should be advertised in the testing squad list). |
| - | * When demoting a package there's | + | * When demoting a package there's an option to keep the current app karma (minor issues) or reset it (big blockers), and a text field where should be added the reason for demotion. |
| + | |||
| + | === Speed Promotion === | ||
| + | * Maintainer can request speed promotion through interface in well defined cases: critical bug in "Extras"-version needs urgent fix; only cosmetic changes (new translations, icons, package description,...); | ||
| + | * Changes must be easily visible/documented (run diff agains version in Extras) | ||
| + | * Speed promotion is done by selected members of the testing sqad. No extra requirements like "10 days" or minimum package karma. [[User:ossi1967|ossi1967]] | ||
| + | |||
| + | === Speed Promotion (alternative) === | ||
| + | Alternate suggestion: given that the smallest change by a developer can cause a serious regression, and there's no way round that - is that once a package reaches the "tipping point" (say, 5 days and 8 votes) another version of the package is let into Extras-testing. | ||
| + | |||
| + | However, people can still rate the earlier version (although not install it) and get it through; whilst the newer version starts its QA process. Obviously if there's a bug, the developer can demote their earlier version and prevent it going through to Extras. --[[User:jaffa|Jaffa]] 13:12, 12 May 2010 (UTC) | ||
== Check List == | == Check List == | ||
<pre> | <pre> | ||
| - | 1. [ ] | + | 1. [ ] Licensing ok. |
| - | 2. [ ] | + | 2. [ ] Description field ok. |
-- | -- | ||
3. [ ] Announced features available. | 3. [ ] Announced features available. | ||
| Line 53: | Line 68: | ||
* UI usability issues cannot be used as a reason for vote down. | * UI usability issues cannot be used as a reason for vote down. | ||
* Always test functionality - that is, run the program and try if it works as it should. | * Always test functionality - that is, run the program and try if it works as it should. | ||
| - | |||
imaginary example: | imaginary example: | ||
<pre> | <pre> | ||
1. [x] Licensing ok. | 1. [x] Licensing ok. | ||
| - | 2. [x] | + | 2. [x] Description field ok. |
| - | FAIL: the package | + | FAIL: the package doesn't have a description. |
-- | -- | ||
3. [ ] Announced features available. | 3. [ ] Announced features available. | ||
| Line 76: | Line 90: | ||
== Testing Squad == | == Testing Squad == | ||
| - | * Can demote packages when there | + | * Can demote packages when there are known blockers. |
* Can promote packages when they are stuck in the testing queue for a while without any known blocker. | * Can promote packages when they are stuck in the testing queue for a while without any known blocker. | ||
| + | * Can promote packages in speed promotion process [[User:ossi1967|ossi1967]] | ||
=== Testing Squad mailing list === | === Testing Squad mailing list === | ||
| - | * Public mailing list where are discussed | + | * Public mailing list where are discussed any situation/issue concerning the applications in the maemo.org repositories. |
| - | * Receives | + | * Receives an automatic notification for each package that enters testing, is demoted or is promoted. |
| + | [[Category:Quality Assurance]] | ||
Latest revision as of 13:12, 12 May 2010
Note: Work in progress, none of the action points below are definitive.
Contents |
[edit] Roles
- Maintainer - the owner of the package under testing
- Tester - Any community member
- Master/Admin - selected testing squad members.
[edit] Automatic checks/Autobuilder
-
Bugtracker field- Done -
That the description field is not empty- Done - Require description field content check only if description has changed
- Optified and dependencies are optified too
- License files included and headers have copyright/license.
[edit] Tool for easy on device testing
- Some easy way to capture a log of power usage, used files and open ports.
[edit] Package Interface
-
Changelog should be displayed- Done - A list of application specific test cases should be displayed (if available. if not available testers should be able to create one.)
- If the package is a library there should be shown packages of application using this library. So everybody can test libraries indirect on application level.
-
Votes should be changeable- Done - Each package that enters or leaves testing triggers a e-mail for the testing squad list
- Link to Wiki so that details of test criteria are always accessible to new testers
[edit] Thumbs Up
- Collapsable check list will appear, testers should mark the fields tested.
- Promotion should occur at +10 karma (if there's at least one completed check list ???)
- or maybe it would be better to have one check per testing task expect for "Working provided features." which should have for example 5 votes. This would be better to make sure that there is no functionality blockers than just one cheking the functionality. (see detailed description in http://talk.maemo.org/showpost.php?p=481536&postcount=55 )
[edit] Thumbs Down
- Testers must comment on thumbs down.
- Maintainers thumb down will demote the application from the testing queue.
- X (fix me) thumbs down will demote the app.
[edit] Demotion
- Packages can be demoted at any time by their maintainers (Implemented) or by a member of the testing squad (demotions should be advertised in the testing squad list).
- When demoting a package there's an option to keep the current app karma (minor issues) or reset it (big blockers), and a text field where should be added the reason for demotion.
[edit] Speed Promotion
- Maintainer can request speed promotion through interface in well defined cases: critical bug in "Extras"-version needs urgent fix; only cosmetic changes (new translations, icons, package description,...);
- Changes must be easily visible/documented (run diff agains version in Extras)
- Speed promotion is done by selected members of the testing sqad. No extra requirements like "10 days" or minimum package karma. ossi1967
[edit] Speed Promotion (alternative)
Alternate suggestion: given that the smallest change by a developer can cause a serious regression, and there's no way round that - is that once a package reaches the "tipping point" (say, 5 days and 8 votes) another version of the package is let into Extras-testing.
However, people can still rate the earlier version (although not install it) and get it through; whilst the newer version starts its QA process. Obviously if there's a bug, the developer can demote their earlier version and prevent it going through to Extras. --Jaffa 13:12, 12 May 2010 (UTC)
[edit] Check List
1. [ ] Licensing ok. 2. [ ] Description field ok. -- 3. [ ] Announced features available. 4. [ ] Working provided features. -- 5. [ ] No performance problems. 6. [ ] No power management issues. 7. [ ] No known security risks. 8. [ ] No illegal/dubious content. Additional comments:
- Put [x] for those tests you have done, elaborate on separate row if the test is FAIL.
- Vote up if there were no FAILs. If there was even one FAIL, vote down.
- UI usability issues cannot be used as a reason for vote down.
- Always test functionality - that is, run the program and try if it works as it should.
imaginary example:
1. [x] Licensing ok. 2. [x] Description field ok. FAIL: the package doesn't have a description. -- 3. [ ] Announced features available. 4. [x] Working provided features. FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123). FAIL: When exporting file the program crashes (see bug: http://url/456) -- 5. [ ] No performance problems. 6. [ ] No power management issues. 7. [ ] No known security risks. 8. [ ] No illegal/dubious content. Additional comments: I liked the program, even though, as is, I have to vote it down due to the bugs. I added few usability enhancements to bugzilla, see http://url/567 and http://url/678
[edit] Testing Squad
- Can demote packages when there are known blockers.
- Can promote packages when they are stuck in the testing queue for a while without any known blocker.
- Can promote packages in speed promotion process ossi1967
[edit] Testing Squad mailing list
- Public mailing list where are discussed any situation/issue concerning the applications in the maemo.org repositories.
- Receives an automatic notification for each package that enters testing, is demoted or is promoted.
- This page was last modified on 12 May 2010, at 13:12.
- This page has been accessed 36,137 times.