[QE] Explicitly specify CRC_BINARY location while running e2e tests

[rohanKanojia]

Component

Quality Engineering

Task description

Description

Related to devtools-qe-incubator/deliverest#59

During our investigation with @lilyLuLiu and @albfan regarding this intermittent issue. We realized that the E2E pipeline on macos-amd64 was testing an outdated version of CRC binary that was present in PATH.

Pipeline copies the built CRC binary to a location present in PATH and assumes that specific binary would be picked up during test execution. However, if someone was using the test instance and created binary in ~/go/bin this binary would take precedence over the one created by the pipeline. This was causing the intermittent issue of pipeline failing due to presence of outdated binary.

To avoid these kind of issues, we should explicitly specify the location of CRC binary as test execution arguments:

crc/images/build-e2e/lib/darwin/run.sh

Lines 53 to 54 in 457b653

	./e2e.test --bundle-location=$bundleLocation --pull-secret-file="${HOME}/$targetFolder/pull-secret" --cleanup-home=false --crc-memory=$crcMemory --godog.tags="$tags" --godog.format=junit > "${HOME}/$targetFolder/results/e2e.results"

Expected Behavior

E2E tests should run with a pinned version of CRC_BINARY

Acceptance Criteria

• CRC_BINARY flag is added to e2e test execution arguments
• Pipeline is tested and it's ensured that correct binary is actually being picked up

[albfan]

While this is desired, explicitly use the PATH for crc binary, we might config a parameter to set it.

If not set, test suite add $HOME/go/bin on top of env PATH

that way we do not harcode any path and allow customizations

[lilyLuLiu]

I suggest to use /usr/local/bin as the default path. As that's where crc is installed during test.