How do you conduct and report the validation testing activities and results?

Ensure that you have in place design documentation and user requirements before your test scripts. Ensure the requirements in both documents are part of the testing.

Along with a comprehensive test design based on the user requirement, a controlled testing environment should set up that closely mirrors the real-world scenarios in which the products will be used. This may involve configuring hardware, software, and networking components. Defining a comprehensive test plan outlining the scope, objectives, resources, test environment, testing methodologies, and success criteria for the validation testing process is crucial for an organised and effective testing.

Ensuring compliance with documented prerequisites and adhering to Alcoa++ principles for data integrity is crucial before conducting tests. Record keeping and execution in line with these principles contribute to reliable and trustworthy results.

Un point très important à traiter lors de la planification est l’identification des risques et les plans de contingence possibles.

The extent and formality of testing should be based on risk. The resulting test strategy covering the types of testing to be performed and the test specifications/protocols to be created, should be defined in a Test Plan. More rigorous and scripted testing should be performed where the impact/risk of failure is greatest. Consideration should also be given to using unscripted testing where the risks are lower. The intention to use unscripted testing can be included in the test strategy and the results of the testing should be summarised in the test report. The evidence of testing can be captured using reports and audit trails in the test system.

Based on my professional expertise, within the framework of current CSA validation methodologies, there exists a notable degree of flexibility in the execution of these activities. A better approach involves the implementation of risk-based assurance by prioritizing requirements and subsequently aligning testing activities with these prioritized risks. An integral initial step in this planning process is to engage all relevant stakeholders, including the Business Owner, System Owner, Quality Assurance team, and Validation team. By doing so, we establish a solid foundation for the avoidance of redundant activities in the later stages of the validation process.

A Validation plan should have a purpose, scope, roles responsibilities, testing approach including vendor testing, how the results will be reported and system will be released.

Something that is shockingly often neglected is when scripts lack sufficient traceability back to specific and testable requirements. For example, for a pathology LIMS, I recently saw scripts within which none of the specific result rules (ie the requirements) being tested were referenced (some PMs think URSs are old fashioned?). While these could often be guessed from the expected outcomes in the later steps of the scripts, that meant testers had to guess what results to enter to get to those outcomes, meaning the scripts also had no repeatability. If you don’t have your requirements in order, you’re not ready for scripting. Forging ahead regardless increases the likelihood of needing remediation, which takes longer overall than GIRFT.

Our report should comprehensively cover all validation activities, typically including: 1. An overview of the qualification documents defined in our validation, noting any deviations. 2. The status of all test cases, indicating how many have passed and how many have failed, and identifying any showstoppers or minor issues. If we are approving the test summary with open defects, we need to provide proper justification. 3. A high-level summary that serves as a single source of documentation, providing a complete overview of what was planned and what was executed in the validation process.

It is worth remembering that system reports and audit trails can provide good evidence of testing and screen shot evidence is not mandatory. It is also important that the tester doesn't just execute the script and record the actual result. Any other observations made during testing of the system not behaving as expected should be recorded. The end goal is not only to report that tests passed, it is to show the system works as expected and is fit for intended use.

During software testing, it is important that the testers executing the test scripts possess proper training and have a thorough understanding of the system. This knowledge helps them with the ability to navigate through the system effectively, enabling them to conduct rigorous testing and identify defects and vulnerabilities within the system.

I keep a video rolling during testing as this is cheap & easy as opposed to not too long ago. Vid caps can be a great way to document screen messages that may not be printable at minimum. Also depending on how liberal your QA team is, its a great way to reduce printing, screen shots, etc. Alternately, perhaps they let you output this form of objective evidence to file format if you (hopefully) are electronic records compliant. Bottom line: video provides BETTER objective evidence than traditional methods b/c you can show that an input is actually LINKED to a specific output. Additionally, this eliminates any speculation that any data has been fabricated. Finally, its more environmentally friendly and inherently storage efficient process.

Leverage cutting-edge technologies and adopt a risk-based testing approach to get maximum benefit during test execution. Rather than confining oneself solely to predetermined test steps, it is highly recommended to embrace innovative testing techniques, such as exploratory and ad-hoc testing. These methods have the potential to unearth a greater number of defects compared to traditional approaches. The means of tracing test cases can vary, such as screenshots, audit trails, reports, and procedural documentation. In essence, your capacity for innovation is the only limit when it comes to achieving superior quality and compliance in the testing process.

Testing should be executed following the pre approved test scripts. A validation incident should be raised for a failed test. Incident should have a clear justification and assess the functionality risk and impact on the business. If impact on the business and risk is low, system can be released for use with a justification.

In addition to the test execution, test environment is an important aspect of validation. The tests should be conducted in the real environment or in a simulated environment. You should be able to justify why your simulated environment is representative of the real one.

When tests fail, the first step is to check the coverage of those test scripts with the project's requirements. Before taking any corrective actions, it's important to assess the severity of the defects. This step helps in prioritizing and determining the appropriate response. The next step involves implementing controls or corrective actions that match the severity of the defect or deviation. For low-severity bugs, it's often acceptable to apply temporary workarounds. In cases where a workaround is not possible, or if resolving the defect is taking an extended amount of time, consideration may be given to deferring certain requirements.

One way to make this step efficient is to bring stakeholders to review the test results summary to decide the next action as soon as it occurs.

This step can be completed as part of executing pre approved test template / test scripts. All the stakeholders that are required to review post execution test protocol /report should be included. This will save time and resources as one less document is required.

If the Test Plan includes the Test Strategy and how testing is to be performed, there is little value in presenting that information again in the Test Report. The Test Plan should be referenced in the Test Report. The Test Report should summarise the actual activities performed, the results, any deviations and how they have been handled. And most importantly include a statement about fitness for intended use. The test records don't need to be included in the report but should be referenced. The use of test tools for recording the test execution and use of reports and audit trails in the system under test as evidence of testing, are valid elements of test reporting.

The test summary report should include a concise summary of the testing process and the defects, bugs, or deviations observed during testing, along with their respective statuses. This summary report should not include test evidence, detailed reports, or corrective actions. Instead, it should reference these documents. Finally, the summary report must conclude by providing a statement on the fitness of the tested system and the recommended next implementation actions.

Drawing from my industry expertise, a well-crafted summary is pivotal in the validation process. When preparing a validation summary report, it is essential to refrain from redundant restatement of the test plan details. Instead, focus on highlighting any deviations from the originally outlined actions. Employing visual aids such as tables, graphs, and reports is highly recommended to convey what was initially agreed upon, what aspects were subjected to testing, the outcomes of these tests, how identified issues or bugs were addressed, and whether the acceptance criteria were successfully met. Strive for a 360-degree perspective, leaving no room for ambiguity, and prioritize clarity throughout the validation summary report.

A Validation summary report is required to summarise all the test reports including any deviations or validation incidents. It should include justification if there is a deviation from the validation plan. Traceability matrix can be included in the VSR which should highlight how the user requirements are met.