Documentation

Introduction

Whether or not your team actively discusses or delegates quality assurance tasks, they're sure to be a part of your process. If you don't have dedicated quality assurance roles on your team, then the responsibility is almost certainly falling on your software engineers, designers, product managers, or all of the above to some degree or other. Worst case, all of your quality feedback is coming directly from your customers.

Testomniac aims to relieve this burden, regardless of where it falls, allowing each person on your team to focus on the primary responsibilities of their role. In the following documents we'll elaborate on the value of quality assurance testing, best practices and principles, and how Testomniac provides a low maintenance solution for your project.

You can also find documentation on this page to help you with onboarding once you begin using Testomniac.

Quality Assurance Principles

Quality assurance is a specialized field, one that is systematic and user oriented. This is not to say that members of your team do not have the right attitude or skills to do quality assurance testing, but ultimately it isn't their primary focus and that tends to introduce risk and gaps in coverage.

A natural conclusion that many teams come to when it comes to safeguarding the end user quality of a product when there are not dedicated QA team members to assign to the task is to have software engineers own building out and maintaining QA automation. There's nothing inherently wrong with this approach, though it naturally conflicts with the core responsibilities of a software engineer: it's both inefficient and disregards the specialized nature of quality assurance processes.

First and foremost, quality assurance should be oriented around the end user experience. A software engineer should be allowed to focus primarily on delivering what was in the spec, making sure that it works as expected, and is built in an extensible way. Quality assurance practices should be focused not only on whether something works as expected, but also whether or not it is a good user experience. Quality assurance is at its best when it functions as a "second set of eyes", offering a different perspective from the person building something or even the person who envisioned or planned it.

There's nuance to this, to be sure, but where Testomniac can step in to fill this gap or take the load off of your team is to focus on the bigger picture. Your team can keep their sights on delivering new features while Testomniac covers your bases when it comes to how the new feature fits into the broader product. With a deep understanding of your product user flows and features, dedicated quality assurance can catch if anything changes in an unexpected way or if a side effect crops up seemingly completely disconnected from a new feature.

One of the biggest benefits of inserting a dedicated quality assurance function and steps into your workflow is mitigating as best as possible the chance that an unsightly issue that maybe could have easily been caught sneaks into production and is experienced by users. The people building the new features should be able to focus on the quality of the code and UI that's being added or modified without then needing to worry about the overall user experience of the entire product.

How to Incorporate QA

When introducing quality assurance as a function or role on your team, the first thing you'll want to be sure of is that there's a deep, comprehensive understanding of your product, from a user perspective. What's less important is an understanding of how the product works behind the scenes, instead the goal is to experience the product as a user does, this being the best way to uncover issues that a typical user might encounter.

In a traditional organization that has dedicated quality assurance roles, a QA analyst would, in simple terms, perform two main functions: verification and regression testing. We'll elaborate in the next section how Testomniac defines the types of testing that it carries out on your behalf, but for now we'll focus on the industry standards.

Verification in this context refers to the process of running through a change after it has been finalized by a software engineer and doing two things: 1) double checking that what was delivered meets the expectations of the specification, or simply put, "does it work" 2) running through ancillary validations, or the less obvious aspects of quality. To elaborate on the less obvious aspects of quality may mean things like: does this make sense? Are there less common or obvious ways to interact with this and do those work? Does this change impact other systems or nearby user experiences? This is the part of the process that introduces the highest risk of slowing down the process, but the upside is that this effectively front loads what otherwise might be churn later on fixing production issues.

Regression testing in this context refers to the process of running through a checklist, or suite of predefined tests. Checklists are a tried and true process, tedious to be sure, but they safeguard against complacency and ensure that no part of your product is ignored for long stretches of time. A regression is when something that worked in a previous build or version of the app stops working, or a bug that was fixed previously crops up again. Regression testing aims to root out issues of this type by running through the core flows of your product, interacting not just with new features but existing ones. This type of testing catches the most nefarious of issues: the ones that you would never expect and are likely to miss.

When we seek to incorporate quality assurance into our processes, these are the primary areas we are likely to get the most value from: a second set of eyes immediately after a new feature or feature update is deemed ready and a user experience checklist runthrough ahead of a production release.

Testing Types

In the above section we defined "verification" and "regression testing" in industry terms. Here we'll elaborate on how Testomniac performs this type of coverage and our specific terminology.

"Ticket testing" is where Testomniac is inserted into your process to perform verification type testing when a change is submitted by a software engineer as "ready". Depending on your team's process, this may be signaled by a ticket in your tracking software being marked a certain status, which is tied back to a pull request in your source control system, or it may simply be the status of a pull request itself. Testomniac picks up the status signals you define for tickets or pull requests, performs testing based on the details provided in ticket and pull request, reports any issues if there are any, and adds tests to the test library based on the new changes if called for. As is the case with a dedicated quality assurance analyst, Testomniac will leave comments on tickets and/or pull requests and update statuses depending on the results of its testing.

"Regression testing" for Testomniac specifically means running through the full suite of tests for the most comprehensive coverage of your product. Although Testomniac tests are built to be fast, this is an operation that isn't meant to be run frequently and instead is done either as a pre release operation or at regular set intervals throughout the week or month, or both. This type of testing is ideal for catching issues that may slip through the cracks, not being obvious at a glance or in parts of the app your team doesn't interact with as much. Without this type of testing, the overall quality of your product can drift in significant ways over time, with little issues compounding and eroding the overall user experience.

"Sanity testing" is a form of regression testing, with a narrower focus. This type of testing zeroes in on the "core flow" or most critical parts of your product's user experience. Sanity testing can be performed ahead of merges into your main branch, offering quick feedback if there's a critical issue that slipped in with the changes. Testing in this way catches what, honestly speaking, could be your more embarrassing misses or trivial mistakes that end up having the most expensive consequences.

At Testomniac, we offer all of the above types of coverage with minimal input from yourself or others on your team, catered to your own particular way of doing things.