Why QA seems to fail | Star Citizen

[@ericvandorp]

Funny, when I learned to program for commercial stuff, my teacher said. If you test it, please test as if the cat is dancing at the keyboard and see what happens. He was right, customers do things you can't even imaginewhen you build a program and if you talk to them, most of the time they say "well I didn't do anything special" and you know they have to pushed a whole lot of buttons before the problem could appear. 😄

[@zeyosin1771]

I'd pay to review a spec sheet and checklist of completed tasks when they implement a new feature to test. Hows the sever performance affect your QA tests? Do they provide an offline server to bypass those bottlenecks? IMHO we backers are the QA without the internal ticketing system.

[@Shadowshade-Gaming]

You know this is a really good idea, I don't know if that's too sensitive according to CIG, but it would give us a lot of insight. I would be interested, but I don't know how many others may be.

[@NoNaNelson69]

pretty sure QA that isnt specifically for net-code is tested on local instances on their PC

[@zeyosin1771]

@@NoNaNelson69 Wouldn't that be counter-intuitive with the fact that almost all actions are server side requests? (doors, drinks, KIOSKs). How do they test? And why not on a PU cloned server with identical hardware?

[@NoNaNelson69]

​@@zeyosin1771 you would be suprises how muhc power the client has in this game - this in not a server sided game - and even if i needs server requests, other games do that too - running server and client on the same machine (e.g. minecraft) it would also be much easier to change server settings

[@zeyosin1771]

@@NoNaNelson69 Actually its a very network reliant system, thats why nothing works when the server begins to struggle. This is the opposite of client based actions/operations that don't exist on server side or are updated at expected intervals. You should research the old quanta framework to better understand what im trying to explain. Testing without the environment parameters is useless when confirming a successful resolution.

[@N1ghtR1der666]

I mean you only have to look at SC's track record of pushing out patches early to know you guys have a hard time getting half of the testing done that is needed before management pushes things out, its over your head and personally I never blamed QA

[@omnipresencetv8929]

They have to get the content out to make sure it works. Because you're part of the alpha process the people who are gonna be buying this later my still be in dippers and to them when the game comes out they might only just be getting the technology to play it in their country. We don't know how many 3rd worlds gonna become the next England or America

[@sebc8938]

The issue with testing automation is that is is usually only possible and easy if the developers have taken it into account in their design. Otherwise you have to test just as a manual user. And if you automate independently from the design, you have to spend a lot of time modifying the testing tools for each update transforming the testing team in a testing tool software team. In the end the issue is also a lot of time that the specification are not detailled enough so the dev thinks they have worked well but there are still a lot of issues in the product.

[@Shadowshade-Gaming]

I really like this comment as I can really relate to it. I will say they seem to have really changed with this year to be more concise and focused. It's just very late in the project to get there and they have a lot to fix because partly of what you stated.

[@MrChrislb86]

I would be very interested to see what their automated integration tests look like. A system this complex, the cost to fix must be very high by the time it hits evocati testing.

[@Thor_Asgard_]

i was in QA in a leading position for 13 years... and QA is not the problem that CIG faces. Their problem is that they are utterly incapable to get anything done, besides timesinks.

[@Billy-bc8pk]

Amazing. Didn't know we had PES and 1,000 player server meshing tests back in 2014.

[@Dulc3B00kbyBrant0n]

@@Billy-bc8pk will those things fix the constant stuttering mess and buggy unplayable trash that is this game? because it doesnt seem logical to assume that everything will somehow work better once they crank up the player count

[@Billy-bc8pk]

@@Dulc3B00kbyBrant0n Watch the tech preview videos uploaded by content creators -- the 500 player servers work better than the current live 100 player servers. The frame-rates are higher, smoother, and everything is more responsive. They have almost got to the point where the 1,000 player server meshes almost work better than the live PU. In two more weeks with proper optimisation, the 1,000 player mesh should be smoothed out. But yeah, check out tenpoundfortytwo's technical breakdown of how much better the 500 player mesh runs over the current live build.

[@dashbolt101s]

Star Citizen is an alpha under development, they are constantly adding new features, gameplay systems, etc. in a game that already combines thousands of different interconnected things. Every single update is extremely likely to bring-in issues with new features as well as old features that worked perfectly fine before. If they took time to do a perfect QA testing phase to eradicate all bugs for each live release, the development wouldn't never be able to move forward. You said it yourself, they don't have enough time. Same goes for what you said about testing with users. Star Citizen actually has a really strong infrastructure for that with the issue council. Internal tests unveil issues but not all of them, that is why every patch then goes into a testing phase with hundreds of Evocati players basically doing the job of QA testers. Then, once major and game breaking bugs are identified and reported, devs do their best to fix it. But even this is not enough, because some issues are sometimes so niche or happening under very specific conditions, even an Evocati testing environement won't unveil them. This is why each build gets into PTU Testing afterward, with thousands of testing players, and then even the LIVE release sometimes unveils issues that where not apparent during PTU testing. Only having internal QA testing for a game like Star Citizen is just impossible, and the current system CIG is using for QA testing is working extremely well to identify bugs and issues and fix the major ones for each patch to be able to provide LIVE release as stable as possibles all while still moving forward with the development of the game.

[@domisPL_01]

@dashbolts101s when bugs remain unfixed then they may cause more bugs later. Imagine building home. You start with foundation level. You are about to start next level but you noticed that part of the foundation is missing. Do you fix it now or after you build next levels over it? This hole in foundation may affect rest of structure above - and will be harder to fix later

[@Shadowshade-Gaming]

You know you are very right, and their QA and developers must be so grateful for the reports of users. Most reports are poorly written, so they still take effort to work on, but literally, the bug council and players are the reason Star Citizen is able to succeed in any form at the moment. If this game ever does get released, it'll most likely have the chance to release with fewer bugs than others because of this. But if it means we as players feel the burnout that the devs and QA feel. Time doesn't only burn funding, but it also kills motivation if you don't have wins. I feel that they didn't get their feet under them until 2021, and I hope it's not too late now.

[@sdwone]

Well OP... It's only been over TEN years!!! What's another 10... Right!? And bugs only get resolved... When someone actually spends the time to resolve them! Otherwise... Bugs on top of bugs will result! And once a hyper complex project hits a certain threshold of unresolved bugs, the only solution left, will be to dump the entire project and start ALL OVER again!

[@PaulMEdwards]

​@@domisPL_01 fortunately, this is software, not hardware. That makes it possible (though not trivial) to completely replace the foundation while leaving the rest of the structure intact. Your analogy is not applicable.

[@PaulMEdwards]

The LIVE PU is a bit of a double-edged sword. It is invaluable in allowing backers to experience the portions implemented (to the degree they are) and provide early feedback. But it's also a liability in a way because CIG has to keep it up and mostly usable while still working on improvements, which definitely slows down progress. That's why the pace of improvements has been fairly slow until recently as more devs have now rotated off SQ42 as their daily work focus and onto SC. I think they were doing just enough to keep us satiated in the PU while they remained focused on SQ42... Out of necessity. It's a big gamble because they're in a "make or break" position where SQ42 absolutely MUST be a significant critical & commercial success, otherwise they risk losing the good will (thus far) of the backers, and moreso the opportunity to gaining more backers in the form of players who've never heard of either game until the SQ42 media campaign gets to them. I, for one, hope they pull it off! 🤞

[@Desenrad]

Can you give some perspective on why SC bug reports so often don’t get addressed? We very often find bugs that seem like they don’t happen to many people. But it genuinely breaks the game for a small group of people. It seems they devs don’t see these as a big enough issue to look into. I see this all the time and it piles up into a buggy mess of a tech demo.

[@Shadowshade-Gaming]

100%, and this will come from a developer perspective vs a QA one. I will start off by saying that from what they have said, the root cause of most of this is from releasing features too early to allow it to be "playable for players". In addition, the developers made poor solutions which were never intended to last this long but are now integral to the game. So it is mismanagement of the development of the game. QA is probably just as mad as we are, but sometimes, when bugs are reported, the issue is far worse than expected. The developer may look at it and see it'll take x amount of hours and involve an overhaul of certain items. In pre-released products, this can be common, but it's now common to have people play or test in pre-release. It then comes to a management choice. Do we fix it now or use "hacks" in the code to make it a little better. Often, the wise choice is to wait until you redo the system where the underlying issue is. Yes, this means it's because one system was written and designed poorly, which often happens when you go fast. If you follow ISC, you can see they have kinda been tackling some of those old bad systems now, which has caused recent releases to take longer, but the overall stability of the game has improved.

[@MrChrislb86]

Its normal that bugs are prioritised, as there is finite time to do stuff. Normally there is a threshold below which fixes are not implemented. Things that influence this are how likely is the bug to occur, is there a work around, how long has it been broken. If it's perceived it will be hard to reproduce the bug (report is vague, or requires a highly specific setup, including a chain of actions), this will impact priority too. If the root cause of the bug is complex, or is planned to be worked on separately in the near future, this will decrease priority. if it's in a specialised bit of code, with only one or two expert developers, there is reduced capacity to take on work.

[@j.d.4697]

Why do you even need an explanation? Isn't it logical that addressing all bugs may not always be possible or smart if the connected systems are getting replaced for example?

[@fathead8933]

@@Shadowshade-Gaming gnerally the major issue is that the dev team is not testing bugs in the conditions in which they exist. They test them on their servers with their optimized system and ping rate. I seriously doubt any of these devs are taking a Walmart gaming laptop to the local mcdonalds and testing the issues in an open live server.

[@Shadowshade-Gaming]

@@fathead8933Yeah, exactly! That's part of what I was hoping to express but I must have not delievered it well. The ability for them to get feedback from us is very valuable to them even if it's frustrating for us. They can try and emulate this via different ways, but the truth is it's very hard to do so. Actually, when we started working from home we started reproducing more bugs at the company I work for since they were then testing with more normal networks.

[@ligmasphere901]

We are QA

[@omnipresencetv8929]

I wanna he fidelity qa but i dont rhink rhey care nuch about fidelity ar rhis point

[@j.d.4697]

You really think just because you did something for many years you know what you are talking about? The people who sit on the couch all day and then make youtube videos about something they never did or learned anything about, THEY are the true experts!

[@I_am_a_human_not_a_commodity]

Typically, if someone does something for many years, they have more experience than the average Joe, so... Yeah, maybe they do know what they are talking about? Your comment really has the same tone as: "You really think just because you might specialize at something you know what you are talking about?"

[@Shadowshade-Gaming]

I feel like there's some sarcasm here and see a joke. I do want to say that just cause you work in a field doesn't make you an expert even if it's for a long time. I played Cello for a number of years and got to play very difficult pieces only because of what my teacher said. Practice does not make perfect, but perfect practice (with intent to improve) makes perfect.

[@I_am_a_human_not_a_commodity]

@@Shadowshade-Gaming I'm just gonna reference Poe's Law and my possible autism lol

[@JS-wt8vf]

I appreciated the content and perspective. Your sound was good and production values were high with video matching what was said. One critique: Your video is very distorted. Almost like your FOV slider has changed from default. This gives it a very fisheye quality and is nauseating to watch and very unappealing. This was less important for this video which largely was about the audio content presented but still made me want to click off the video multiple times. Keep up the great work!

[@Shadowshade-Gaming]

Thanks for the feedback!

[@glalih]

I have a friend that wouldnt give a QA pass for the loading screen on star citizen. QA is nonexistent, they just covering their asses for features they lied about 10 years ago.

[@AndrewJdotcom]

These video's you make about QA are pointless when it comes to Star Citizen. I wouldn't say this game was in early access or even a Beta in the sense most people are familiar with these day. I would say it's more like your being allowed to play a game that's in still in early-mid development, expecting them to test and fix things that are constantly being changed is a exercise in futility and a waste of development time. The sooner people realise this the better. Talking about QA closer to it's completion would make alot more sense, which will most likely be many years in the future unfortunately. It might be a hard pill to swallow for most people but that is the reality. Just be glad there's at least one development studio out there trying to make a truly next-gen game and you get to at least have a peek behind the curtain in its early stages.

[@Dulc3B00kbyBrant0n]

but thats the thing is they whine about everything , server costs, their own backers, the public themselves who cant grasp the ways of the high brow roberts industries above the entrie gaming industry itself, meanwhile theyve been paid already for the game they never made and have people testing it for free... the worst part is that the game is getting worse for everyone even the whales because they cant even play a game that doesnt break every 5 minutes get crime stats in their own hangars getting their own ships impounded is just the beginning of the broken fuckery that is this game at one point CIG themselves are incentivised to leave the buggy mess as is because it encourages demoralized players to buy ships or just stop playing ultimately if they dare grind their way up then its win for CIG because they get off the servers... just the palpable feeling you get when you login and u see ur stupid health meter going down in real time because they HATE having you play on their servers... what dumb game

[@AndrewJdotcom]

@@Dulc3B00kbyBrant0n They probably would prefer it if no one was playing the game. Then they could focus on just making it and not catering to to hordes of players that don't understand what it is they are playing and paying for, The problem you have then is (and I can't think of an alternative way, maybe you can?) how do you fund something on this much of a technical scale? Because no other studio is going to take the risk with their own money, what they are doing has never been done before and there is a chance it might not even be possible for a long time and I can gurantee you no one else is attempting anything like this. So i hope you like games with no innovation like Call of Duty because thats what you will continue to get for many years to come if this fails.

[@takeonedaily]

Let's stop the charade, we all know the backers (current players) are the real QA.

[@-MaXuS-]

Forgive me..but “your company”..you can’t be older than 16-18 years old. What kind of company and how many employees does this supposed company of yours employ?