Thread by @JayMFernandes

Since I'm submitting my talk and in honor of the 50+ games I've worked on across pretty much every modern platform you can think of:

1 like = 1 piece of advice for releasing your game that I learned (feel free to share for more answers).

I'll also take specific Q's

1. For consoles, don't underestimate the tools that platforms provide & PLEASE spend time familiarizing with how they function, it will save you so much time in the long run. The dev forums are your friend (and a lot of times I get asked on each game how to do xyz on a kit)

2. Do your QA early, yes, srsly, I know you might be 'still designing' or in the process of doing other things but you benefit from testing and processes still. I've worked on games from prototype phase all the way to post-release (you just look at diff things at each milestone)

3. Your 'release' phase (trying to submit your game or finish it to release) is going to take longer and cost more than you think, plan for contingencies and 'plan b/c/d'. NEVER assume someone else will always be on time with their delivery or feedback (including platforms)

4. Hire an expert when on a platform you're unfamiliar with, have them look at your game & discuss potential issues/benefits that platform may have had to delay games, modes, & functionality on bc I was brought on late and pointed out issues after work had been put into them

5. Make sure your schedules are realistic and line up correctly with everything you need delivered (or delivered to you). I've seen games where 'submission' was before final art or code freeze or was set before the game would even be considered out of 'Beta'.

6. QA isn't just testing, my job is guiding devs through varies processes to help them save money + make the best use of their time (or to hit their seemingly impossible dates with as little risk possible). It's education, consultation, and more

7. Continued: My job is more than QA. UI and other things been added under my suggestions to make the player experience it's best (yes, I am a designer among other things). Thebenefit is usually less bugs + happier players too. You've definitely played something I made/suggested

8. When budgeting for how much to put aside for testing in your project, scope plays a huge part. Some things also are inherently more (like multiplayer). MP also is never *double* the cost, it's usually 3-4x that just for testing + more if different platforms are involved

9. I'm accurate at test hours, I've done 3000+ hr projects that were >1 year that ended ~80 hours off my orig. estimate. I do it by taking everything into account + planning for worst case/contingency (how long for submission, what modes, how big, how long etc) and extrapolating

10. More ppl =/= finishing faster (this is the dev hour myth). 1 person working for 2 weeks is not the same as 2 people working for 1 week (esp if you have to train them) which is also why earlier integration into teams makes for more efficient and effective work in general

11. Some platforms can literally be more work than others because of their intended design for player experience so DO NOT assume that you can straight port your game to all of them with no platform specific changes or user flow involved. This will bite you in the butt

12. Porting to consoles is cheaper than it used to be but is still more expensive than their PC counterparts. Not only can kits cost, but there is dev time, QA time, release management and other things to consider that end up making them unattainable for many.

13. Hiring QA/UX or others can become practically useless if you dismiss them/their advice. I cannot tell you how many times I've talked to developers who dismissed me/my team then came back freaking out about why they are failing their submission or why players are having issues

14. Communication is extremely important, especially when trying to reach the finish line and getting all of the pieces together. If there is a miscommunication, depending on what it is, it can (and I've seen) cause failures, missed dates, and massive budget issues trying to fix

15. The biggest things I see people fail console submission for are usually fairly simple guidelines that were missed or ignored. A lot of times it's text/branding related or something consoles point out in their standards (each has a 'top failures' list, use it!).

16. You don't need a publisher to have good QA on your game and what publisher's used to provide has changed over the years. Make sure you understand what's being provided (or not) and plan accordingly. Most games I worked on were either sans publisher or working for one directly

17. My least favorite platform to work on is mobile, mostly for device and screen compatibility reasons. PC can also have similar issues with all of the processors and cards out but for mobile it just seems so much worse to me. There are companies to do device compat for you :P

18. Sometimes an unrelated thing can cause issues you didn't think about. A game I worked on would crash eventually when a Kinect was attached (game had no Kinect support) because it turned out the engine had issues with it + was causing the bug. Had to wait for an engine patch

19. Part of my job can be to help developers figure out a best course of action. I've been on calls with studios + platforms talking about release schedules and risks etc. and have been able to help facilitate solutions that worked for everyone. (ie should we submit now or later)

20. Releasing is one of the hardest parts, & it's very difficult to take on everything needed by yourself. If you have a team make sure you're utilizing them (+ thinking ahead for needs). There's not only a game to finish but marketing materials, socials, community, patches etc.

21. QA takes time, both literally and figuratively. Don't assume you'll find everything in 3 days if you put a bunch of people on it. Allow time for exploration of the game's systems and time for edge cases to come up. If a tester gets a bad bug, you know users will too

22. QA can be a way to get early feedback on the game as well as insight into how players will use it or understand what you've designed. My team specifically also has training in other areas so feedback on usability and audio usually come with it (which makes for a better game!)

23. Playtesting can be v important for any type of game. If it's a multiplayer game where balancing is also needed? Ask testers how they feel about certain things, like any feedback be wary of pitfalls, but this can be a good litmus test (since they probably play more than you)

24. It's ok to cut things. Really, sometimes you need to! Maybe that new thing doesn't work well enough compared to the rest of the game, isn't as fun, or feels 'tacked on'. Making sure the final product matches the intended design for players can involve scaling waaaay down too

25. DO NOT wait until the last minute to add content that's never been in before (especially if you're on consoles) since there's a good chance it won't work as designed or that it could be better through iteration. Assuming it'll work fine the first time is... not ideal!

26. If you're planning to fix things through patching be aware of how the timelines work for each platform. Some require more time to verify patches than others so a 'day 1' or hot fix on PC could take weeks on console, plan accordingly!

27. Early Access and beta testing is not a substitute for trained/specialized QA/UX testing since you're missing the experience and communication of experienced testers/researchers. Bugs you get from each group are different and need to be treated differently

28. You can use Early Access/Beta testing as a way to get feedback from a larger pool of players and to 'out' the more serious surface level issues, but generally those types of players will not catch issues that are compliance failures or more complex functionality bugs

29. Besides pointing out issues, I also do a great deal of working with devs on what an ideal solution would be (ideal as in would satisfy platform, dev, player, budget, time expectations). Having an experienced dev who knows the platform helps more than random players

28/29. Caveat: There is a great use for EA and player feedback, but just understand that it's different than feedback I would give as a developer from a QA/UX perspective. Player feedback can be extremely valuable when parsed correctly!

30. Debug tools are the greatest things ever when used correctly in test. Ask your team if they need anything and be open to suggestions if they have a need, on some games this can save literally DAYS or WEEKS of work using debug instead of playing 'normally'

31. Task tracking tools (basecamp, trello, etc.) are not the same as bugtracking tools and serve different functions. Even though you can use one for the other, you're trying to shove a square in a round hole and not able to utilize the tools to better your process in the future.

32. Games get better the more I work on because I utilize processes to look for the 'core' cause of an issue during test/release and fix it moving forward. Bug tracking allows kpis to help with this (like say, how many times a bug was claim fixed before actually being fixed)

33. Automation is not the silver bullet for solving test/QA issues and probably never will be. You can automate some testing but many things in games still need manual testing done, work with your team to see what makes sense to automate (cost vs benefit)

34. One of the biggest reasons for failures during submission to platforms is either recent changes that have not been looked at enough or new additions that weren't present during the rest of the test cycle. Give time to make sure fixes are stable/correct

35. Making sure things are compatible is a huge undertaking. Different controller types, audio output, video output, and other things makes this a resource sync for modern games utilizing those systems, make sure they all actually work

36. It's a bad idea to only have generalists in QA. I catch audio issues others don't know exist (5.1 channels switched, comb filtering, incorrect placement of items in 3D space etc.), Compliance, compat, network, and other specialists exist as well and round out teams

37. QA should be a part of your design meetings. It helps everyone get on the same page about intended design of systems as well as the time needed to test those (as well as develop them since ppl usually forget the former) as well as allowing for discussions of how best to test

38. Sometimes you'll get bugs that are opposite your intended design from QA. Generally this is bc of either 1. they do not know the intended design/it's not obvious or 2. your intended design looks like a bug (and others/players might think the same). Think about which it is

39. 'Simple' games do not automatically mean simple to test. This is why it's good to have QA input on what is needed to finish the game to the level you want and so people understand the end-goal.

40. For indies, the Q isn't usually 'how much testing for the game to be perfect' it's 'how much testing can I afford and what's the most useful'

Costs add up quite quickly so it's important to be smart about where your resources go to. It can be a never-ending service

41. I joke about being able to test Skyrim in a day because it's a service where you can pick 2 :

-Good
-Cheap
-Fast

So I can do something quick and cheap but it won't be that good. Make sure to strike that balance between the three for what makes sense in your studio

42. It's largely a logic puzzle as well, where you can't guarantee completely 'bug-free' but there are things you can do to make it as low-risk as possible (given the time/budget). I support a risk-based methodology where areas are checked based on a moving target of 'high risk'

43. When calculating costs everyone always forgets the admin side of releasing a game. There's time needed to research, write, report, chat w/platforms, write test cases/materials, and other things that are generally forgotten. No tracking means no ability to learn going forward

44. Console compliance standards and versions you can use for things change a lot and it's important to stay updated. Make sure you're looking at those so that when you submit you're on the versions they 'allow' and that your compliance standards are up to date

45. PC compatibility is generally cheaper to outsource or couple with beta/Early Access type playtesters then it is to buy the equipment yourself

46. A common misconception is that games need to functionally work 100% to be released on consoles but that's not exactly what they are looking for in format QA. Generally only "critical" functionality issues would prevent a release, *if* they find it (think crashes).

47. Sometimes testing is working from specific checklists (usually compliance) other times it's going off your gut and testing what you think will break first (ad-hoc/exploratory). There's pros and cons to each and partially why bugs get missed had to do with the "system" you use

48. The ideal QA process in a studio is one that is proactive instead of reactionary. This means creating processes to prevent errors from occurring in the first place as opposed to having systems just to find and react to errors

49. Documentation is really helpful for QA if you're hitting ones unfamiliar with the game. It allows them to know how systems should specifically function as well as know of any secrets or hidden petitions to make sure they work. Doesn't need to be a whole gdd but an overview

50. In general it's bad practice to use debug all the time when testing. I've seen trophies that don't unlock naturally, items that don't exist in game, levels that load on debug but not playing naturally. Don't solely rely on it

51. QA processes and priority for issues can vary greatly between companies so when hiring make sure you are clear about yours. Some value player experience, or accessibility, or other things over bare design functionality verification. Others are just straight 'does it work?'

52. If you plan to have your own internal QA team, plan far ahead. What equipment they need, permissions, who gets final say on issues. ESPECIALLY think about what they do if there's no testing to be done or once the game ships, do they do other things? Or laid off? (Boo)

53. Testing is as much about being 'good' at the game as it is being 'bad' at it. A lot of issues stem from dying, restarting a lot, backtracking, moving through the level 'wrong' or skipping things, pretty much playing like someone who doesn't know the game well. Include this

54. If you've never used PICT it's rad and can be super helpful for large matrices that you can't test through. I've used it a few times to help with random combination testing to be more efficient and not double up github.com/Microsoft/pict

55. Macros can also be helpful tools for testing things (like Logitech's built in ones or other 3rd party things). I've had macros to do 15 step debug functions or common shortcuts I use testing games. They also come in handy for doing repetitive tasks in games (collect coins etc

56. Sometimes bugs are missed, it happens. Accept it and move on by figuring out how to prevent that slip going forward and making sure it's checked on future patches/titles. Don't bother placing blame on individuals unless it's *literally* their fault, it's probably bad process

57. The failures that occur during submission are extremely valuable. We save and track them so that those failures can be checked on other titles moving forward, which means our submissions get better the more we do. Last team we averaged more than 60% first pass rate

58. Just bc a platform didn't find a bug you know fails the first time they looked, doesn't mean they won't find it the next time(s). It's still lower risk to fix those issues than 'hope' they won't see it. Cleanish first pass with 1 fail can turn into 5 fails the 2nd pass easily

59. Make sure you have your ratings, bank info, and other items needed by platforms before you're in the weeds trying to navigate the release cycle. Lots of times 'small' things like this can turn into big delays

60. If your game has a lot of text or voice dialogue it's really good to get test a copy of those lines and where they occur so they can catch missing ones (happens a lot). If you're unfamiliar with something it's not always obvious if a line was missed or if it's awkward silence

61. For compliance testing on consoles, I rarely ever like to have only one set of eyes on it. This is because many things can be subjective or taken in different contexts so having 2 people looking at it to discuss helps a lot. Our team is great at this imo

62. Some of the platforms offer per-certification type testing that does a dry run of the compliance checks, take advantage of it! It can tell you a lot about where the build is currently at and be a litmus for when you're 'release ready'

63. QA and CS usually work pretty closely together in larger companies. Partially this is bc QA would (probably) already be aware of issues that players would get (and thus CS should know about). It's good to share what issues you're still concerned about for release, advocate

64. Your test team should learn other skills too, since it helps make better QA anyways. Since I started as a tester I've learned audio, programming, art, animation pipelines as well as how major engines 'generally' work when developing and it helps me recognize issues quickly

65. A fresh set of eyes on a game can be really good. Newcomers to the team can out issues you're blind too quickly as well as provide fresh perspective into it's current design + function. If your QA team has been on the game for a long time, new tests can also help out old bugs

66. Testing a multiplayer game (say 4 players) with 1 person on 4 consoles is not the same as testing with 4 people on 4 consoles and you will find different issues. The latter is closer to real world while the former is more like how you treat debug testing

67. Polish will take longer than you think. No really, it's longer than that. Most games ship in the middle of the polishing phase because of other constraints but it's so important to make time for it because it really helps sell to players what's so special about your game

68. Testing on a port has it's benefits like how the internet probably already made a wiki with exactly how and where to get everything you need (instead of having to build your own). plus you can check functionality in the original

69. Testing on Mac and Linux is generally costly compared to the ROI you'd get from sales alone (depending on the game). It's a balancing act of customer support as well since those are generally much more time consuming to troubleshoot, but can be very worth doing for players

70. A common tactic when releasing a game on platforms is to have the 'base' game ready enough to release with a 'day 1' patch already being worked on before the title is even released. This can be good if you need to make a date, but also puts more strain on your qa and dev team

71. (cont'd) partially this is because now teams need to remember which branch has what fix and make sure that they are communicating and tracking fixes for these correctly. Sometimes, the base gets a fix to pass submission (but it's not the 'patch') so make sure to keep tabs

72. There are unarguable benefits to adding languages into your game, but besides the cost of translation there's also the cost of making sure they appear correctly in game (and that trophies and other things work). It also means playing through in each language generally

73. Since random seed and other types of procedural type design is popular, it's helpful to have somewhere where testers can see the seed used and (even better) be able to input their own in order to test fixes for issues they find

74. If I write and issue that is waived by production and ends up failing in submission, I'm not bringing up the issue bc I want to say "I told you so". I bring up the issue so we can look at how long we knew about it and base future decisions on that error/cause, for next time

75. It's a REALLY bad idea to submit to platforms and use them as a sort of QA team to write issues on your game like a traditional one would, it's not their job, and it wastes resources. The difference is they will stop testing if it's too bad, and only write issues in order

76. A traditional or internal team will write all of the issues at once as they find them, while platforms are only writing ones they see in their tests and will stop tests if there's blockers, only to write more issues once that *one* thing is fixed. You'll waste your time

77. Be respectful, even if they aren't on your team normally. If I'm bringing up that I think a schedule or something else may seem aggressive or makes me uneasy, I'm speaking from experience and bc I'm worried the game won't ship on time. It's not a personal attack

@threadreaderapp unroll plz