Onboarding (QA Testers)

This guide is intended for QA members to understand the overall QA process. If you are not a QA member and just want to know how to report bugs, refer to the simplified Bug Reporting guide here.

Quick Links

• Build/Tool Access Form: https://forms.gle/uaxbtcRwwCzQnX4z7

  • All fields are required for QA members.

• ClickUp: Coding Board

  • Clickup Kanban boards are used for tracking coding QA tasks and feature testing.

• Notion: Level Design/Navmesh, Landscaping (QA only)

  • Notion boards are used for tracking interior and exterior QA tasks and testing

• Github: https://github.com/TESRSkywind/QA/issues

  • Github Issues is used for issue reporting and tracking.

Installing Skywind

The latest Skywind build can be found on the Discord server in the #builds channel. Current installation instructions can be found here: Installation Guide

• Start with a clean install of Skyrim Special Edition. You will need to install SkyUI and SKSE separately, but to ensure accurate testing, do not install any other mods. If you have the Anniversary Edition, you will need to downgrade to the non-AE version.

• Always test with the most recent build. Install the most recent full version first, then any patches, hotfixes, loose files, etc. afterward, in chronological order to ensure you have the correct, latest files.

• Merges happen on a roughly monthly basis and are announced ahead of time. You'll want to download and install as soon as possible after release, but be aware it's common for there to be 2-3 hotfixes within a week of release!

• A full download is about 18 GB zipped and 30 GB unzipped. Be sure you have enough space before trying to unzip or your computer will have an unpleasant time.

  • Update builds tend to be around 1-2 GB, updated LODs are around 1 GB, and hotfixes are usually only about 100-200 MB.

  • The updated LODs are technically optional, but without them, your LODs (models seen from a distance) will be out of date.

Clickup Overview

Clickup is used for overseeing development pipelines, including QA tasks and feature testing.

QA work on Clickup is tracked for Coding (Quest testing) only.

Coding Only: Navigating the boards can be challenging due to the large number of columns. The workspace filter "QA Columns" can be used in these boards to limit the view to the boards described below:

Board Columns

Ready For QA/Ready for Retest: These are cards that are ready to test! When you’re ready for a new card, you’ll move it from this column to QA In Progress. All cards in this column should have a checklist attached (see card description). If a card is missing its checklist, contact one of the QA leads. 

QA In Progress: These cards are actively being tested by a QA person. All cards in this column should have a member assigned. When finished testing a card, you’ll either return it to the landscapers or move it to Testing Blocked or QA Reviewed as appropriate.

QA Reviewed: These cards were tested by a QA member and issues were identified. Someone from coding team will address the identified issues and send the cards to the Ready for Retest column.

Complete: These cards work perfectly and require no further testing! With luck, we’ll never look at them again. (If we need to reopen them, a QA lead will move them back to Ready For QA as appropriate.)

Notion Overview

Notion is used for overseeing development pipelines, including QA tasks and feature testing.

QA work on Notion is tracked for interior and exterior testing only: Level Design (Interior testing) and Landscaping (Exterior testing).

Landscaping Only: Navigating the boards can be challenging due to the large number of columns. The "QA Tasks" tab can be used in these boards to limit the view to the QA boards. Alternatively, you can sort the status of tasks by selecting the "status" column header in the "Show all" tab and select the type of tasks you are looking for. You can see both options below:

Notion Automations

The task status' in notion are the same as they are in Clickup so the same rules apply. 

Notion however has some automation buttons that make accepting and sending off tasks very easy. When you select a card in the QA Tasks tab, you will see action QA buttons as you scroll down towards the bottom of the card. These buttons will automatically send the card to the proper columns. If you are in the "Show All" tab, these buttons are on the same row when you scroll further to the right of the page. 

Selecting the "Begin QA" button will add you to the card and send it to the "QA in Progress" column and change its status accordingly. 

"QA Passed" or "QA Failed" will send the card either to the Complete column, or back to Level Design for fixes.

Feature Testing Process

This process is for testing complete feature cards in Clickup.

1. In Clickup, move a card from the Ready For QA or Ready For Retest column to the QA In Progress column. (alternately, you may be assigned a card by a department lead). Make sure that the card contains a valid link to a checklist in its description. If this is not the case, notify a team lead before continuing with the testing process.

2. Assign yourself to the card and add a due date.

   1. The due date is not a hard-and-fast date, just whenever you think you’ll have the card fully tested by. This is important so we can see what people are working on!

3. Begin testing the card. Use the Landscaping and Quest testing guides in the Instructions/Resources column for information on what/how to test.

   1. If there are Github issues attached to the checklist, be sure to test these as well. Be aware that other departments sometimes close issues before they’ve been retested, so retest any issues that aren’t labeled Confirmed Fixed.

   2. Be sure to read the appropriate page on UESP to get an overview of how the quest should work, what sort of loot to expect in a location, etc. Keep in mind that some quests and locations have been rewritten or significantly altered; if you're unsure, ask about it!

   3. Fill out the Checklist on the card. Do not change the share permissions on the checklist!

4. If everything is working properly and all issues are fixed, add a comment to the card saying that testing is fully complete and move the card to the QA Reviewed/Complete column. Only do this when a card is 100% functional, with no issues or unfinished pieces.

5. If the card still has issues, open or comment on Github issues as necessary and add a comment to the card linking to any new Github issues. 

   1. Coding: Once you've added the Github issues, move the Clickup card to the QA Reviewed column, which will indicate testing could not be fully completed.

   2. Landscaping:  Once you've added the Github issues, move the Clickup card to the Review Returned column.

6. Remove yourself from the card after testing.

Github Overview

Github is used for reporting new issues (primarily by QA, but sometimes by other team members), tracking progress in fixing issues (by other departments), and retesting fixed issues (by QA). It is not used for downloading the build--we tried to make it work and had a boatload of problems. It is what it is.

Because Github is used by multiple departments, you can see an overview of it from a non-QA perspective here.

As QA, we’re expected to have a higher standard of bug reporting than other departments, who may not be familiar with the testing process! So here’s a more in-depth look at what happens with bugs: 

1. When you find a bug, search Github to see if the issue already exists. If it does and you have additional information/screenshots/etc., add them to the issue.

2. If the issue does not exist in Github, create a new issue using  one of the issue templates: https://github.com/TESRSkywind/QA/issues/new/choose.  Add relevant department labels if missing. If there is a related Trello card, add a link to the issue to the card.

3. The QA leads will review issues with the New Issues label. If they’re determined to be duplicates, invalid issues (e.g. something simply is unfinished, not actually broken), too vague, etc., they may be closed or have more information requested. If the issue appears valid and has enough information, it will be labeled as Confirmed.

4. The issue is now the responsibility of other departments! They will review issues and either close them, fix them, or request more information. (if they request more information, please answer them!) When an issue is complete, it’ll be marked as Claim Fix.

5. Once a build including the fix has been released (has the In Current Build label), an issue is ready to be retested. If you’re retesting a particular feature card on Trello, you’ll also be retesting any linked issues; for “independent” issues, I’ll ask people to test them directly.

6. If you retest and the issue is not fixed, remove the Claim Fixed label and add the Fix Fail label. Add a comment saying which build you’re using and any additional details. (If the issue was closed by the person who worked on it, reopen it.)

7. If you retest and the issue is fixed, add a comment stating which build it was fixed in. Then use the "Close Issue/Close with Comment" button to close the issue. This will add the Closed Fixed label to the ticket and remove the Claim Fix label.

Regression Testing Process

This process is for retesting specific issues that have been marked as Claim Fixed and In Current Build in Github. They may or may not be tied to a feature in Trello. (Quick Link)

1. Assign the issue to yourself in Github. If it’ll take awhile to test, add the In Progress label to let others know you’re retesting it. (once you’re done, of course, remove this label)

2. Retest the issue thoroughly, which may mean going beyond the scope of the original bug ticket.

3. If the issue has not been fixed, remove the Claim Fix label and add the Fix Fail label. Add a comment saying which build you’re using and any additional details about what hasn’t been fixed, if it’s partially fixed, etc.

4. If the issue is fixed, add a comment stating which build it was fixed in. Then use the "Close Issue/Close with Comment" button to close the issue. This will add the Closed Fixed label to the ticket and remove the Claim Fix label.

Closing Invalid/Erroneous Tickets

Make sure to use the Closed as not planned button if you need to close an invalid issue. You might have to click the drop-down button next to Close Issue  in order to access this button.

Labels

Github labels are used to organize issues. They generally fall into one of two categories: status labels and department labels. We don’t use all the labels, but here’s an overview of what the major ones mean:

Status Labels:

• New Issues - just been created, but hasn’t been reviewed yet (add this to all issues you create!)

• Confirmed/Reviewed - reviewed by the QA leads and ready to be worked on

• In Progress - actively being worked on by a landscaper, coder, tester, etc.

• Claim Fix - a fix was completed but the issue has not been retested

• Fix Fail - retested; not actually fixed yet

• Closed Fixed - retested; confirmed successfully fixed

• In Current Build - fix is present in latest build

• In Pending Build - fix has been made but is not yet in the released build

• Invalid Issue - issue cannot be reproduced, is a result of unfinished work, isn’t actually a bug, or otherwise doesn’t need to be fixed

• Known Shippable - will not or cannot fix (hopefully not common!)

• Duplicate - issue already exists but was logged again in error. (Once an issue has been marked as a duplicate it should be closed as not planned.)

Department Labels:

• 2D - any 2D assets such as banners, letters, books, etc.

• 3D - problems with textures, models, tilesets, etc.

• Animation - animation of creatures, NPCs, combat/spells, containers, etc.

• Audio - music, creatures, armor/weapon SFX, weather SFX, etc. (use Voice and Dialogue for VA-specific issues)

• Coding - NPC behavior and schedules, skills and abilities, incorrect item enchantments etc.

• Quest - all quest related issues (do not includes standalone item/level design issues)

• Implementation - problems with rigging of armor/clothing/weapons, model collisions, and the like

• NPC Appearance - problems with NPC clothing, equipment, voice/face mismatch etc.

• Landscaping - exterior terrain, clutter, navmeshing, shaders, lighting, model placement, clipping, etc. For exterior landscaping only.

• Level Design - interior terrain, clutter, navmeshing, shaders, lighting, model placement, clipping, etc. For interior landscaping only.

• UI - inventory, leveling/perks screens, fonts, character creation, loading screens, dialogue, quest journal, etc.

• Voice and Dialogue - voice acting, subtitles, dialogue, etc.

• Writing - narrative text (e.g. newly added books or notes), inconsistencies in quests/dialogue, lore problems, etc. 

Frequently Asked Questions

See this page for some frequently asked questions about QA.