Squish test functions always pass when used with squishtest module

Question

We are using Squish for Qt 6.6.2 on Windows 10 with Python 3.8.7 and running our tests using squishtest module with Robot Framework 4.0.1.

We are having an issue with the test functions provided by the Squish API where any verifications done with such a call (for example squishtest.test.imagePresent) will Pass. The issue itself was quite simple to pinpoint to the fact that although the verification failed, the function call itself was passing without raising exceptions. This can also be verified from the report provided by the squishrunner where we have <scriptedVerificationResult type="FAIL" time="--"> on the passed execution.

The question is, can we in any way get the actual verification result passed to the Robot so we can fail the test accordingly? Preferrably in real time rather than parsing the report afterwards.

In Squish this works perfectly fine

def main():
    startApplication("AUT")
    snooze(2)
    test.imagePresent("image.png", {"tolerant": True, "threshold": 85}, 
    waitForObjectExists(names.sceneContainer_GraphWidget))

but with Robot this is always passing

# In testSuite.robot

*** Settings ***
Library     MySquishLib

*** Test Cases ***
Test Image
    Start AUT
    Verify Image    image.png    {"tolerant": True, "threshold": 85}    names.sceneContainer_GraphWidget

# In MySquishLib.py

import squishtest
import names

def start_aut():
    squishtest.startApplication("AUT")

def verify_image(imageFile, imageParams, imageArea):
    squishtest.test.imagePresent(imageFile, imageParams, imageArea)

test.imagePresent() is not documented to throw an error. For me, your low tolerance threshold causes the addressbook main window to be "found" even when it is actually not on the screen. I cannot reproduce the problem while using the squishtest module. test.imagePresent() returns true or false as expected for me. Can you reproduce the problem when using squishtest without the Robot framework? — frog.ca
– frog.ca, Commented Sep 27, 2021 at 11:04
My wording may have been a bit unclear, but that was exactly the point of this question. I would like the function call to either A) Return the status of the VP or B) throw error on failed VP. At first I expected the tolerance to be an issue as well but as mentioned, in the results.xml created by Squish, the VP has passed/failed just as expected. — Morkkis
– Morkkis, Commented Sep 27, 2021 at 11:41
@frog.ca The "issue" was actually in the fact that, as you can see from the question, I'm actually not using the return value anywhere. As you pointed out - The function does return bool value correctly and is not supposed to raise exception. Additionally, the outcome can be found during runtime from the results.xml. If using xml3 report, the values can be already accessed there before the test ends. — Morkkis
– Morkkis, Commented Sep 27, 2021 at 11:59

Eugen_R · Accepted Answer · 2022-10-26 08:50:29Z

1

Have a look at the documented: bool testSettings.throwOnFailure flag. In your robot you can set this and you would not have to patch / rewrite every test.vp, test.compare, ... method.

When running from within the Squish IDE, this flag can be unset. Probably robotframework provides some environment variables to detect wether the test case is running from within itself.

https://doc.froglogic.com/squish/latest/rgs-squish.html#rgs-testsettings

answered Oct 26, 2022 at 8:50

Eugen_R

795 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Morkkis · Accepted Answer · 2021-09-27 12:03:32Z

0

The test functions are not supposed to raise exceptions during execution in order to allow test to continue even if a single VP was failed. The function does however return a boolean value just as expected. By using

def verify_image(imageFile, imageParams, imageArea):
    if not squishtest.test.imagePresent(imageFile, imageParams, imageArea):
        raise Exception("Image was not found")

I'm able to fail the Robot test without any issues.

answered Sep 27, 2021 at 12:03

Morkkis

5053 silver badges13 bronze badges

Collectives™ on Stack Overflow

Squish test functions always pass when used with squishtest module

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related