Cli tests - synchronous testing of platform commands

[JosephGJ]

Adds tests for the amp platform commands:

• pull
• start
• stop
• status
  (Monitor can't be tested as it does not return to the cli)

Tests are implemented in the setup and tearDown directory platform.yml files. The setup and tearDown YAML test specs are run synchronously to the samples, with setup being before and tearDown being after. In addition, the docker swarm is initialized in the setup and left in the tearDown.

New field added to the YAML file - run. Boolean which says if a command should or should not be run. By default, the commands in setup and tearDown are not run because they can take upwards of 5 minutes to run.

To test, change the run fields to true in the platform.yml in both setup and tearDown directories. Ensure your platform is stopped and run docker swarm leave --force (Or you can reset the docker client), then run:

go test ./tests/cli

[neha-viswanathan]

The tests are failing -

=== RUN   TestCliCmds
--- FAIL: TestCliCmds (18.26s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x80d3e]

goroutine 5 [running]:
panic(0x18dd40, 0xc42000e060)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
testing.tRunner.func1(0xc420086180)
        /usr/local/go/src/testing/testing.go:579 +0x25d
panic(0x18dd40, 0xc42000e060)
        /usr/local/go/src/runtime/panic.go:458 +0x243
github.com/appcelerator/amp/tests/cli.runCmdSpec(0xc420086180, 0x1, 0xc4200b6080, 0x11, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/nviswanathan/dev/src/github.com/appcelerator/amp/tests/cli/cli_test.go:207 +0x95e
github.com/appcelerator/amp/tests/cli.runFrameworkSpec(0xc420086180, 0xc420014960)
        /Users/nviswanathan/dev/src/github.com/appcelerator/amp/tests/cli/cli_test.go:136 +0x172
github.com/appcelerator/amp/tests/cli.runFramework(0xc420086180, 0xc420028098, 0x1, 0x1)
        /Users/nviswanathan/dev/src/github.com/appcelerator/amp/tests/cli/cli_test.go:114 +0x4e
github.com/appcelerator/amp/tests/cli.TestCliCmds(0xc420086180)
        /Users/nviswanathan/dev/src/github.com/appcelerator/amp/tests/cli/cli_test.go:90 +0x571
testing.tRunner(0xc420086180, 0x1d0158)
        /usr/local/go/src/testing/testing.go:610 +0x81
created by testing.(*T).Run
        /usr/local/go/src/testing/testing.go:646 +0x2ec
exit status 2
FAIL    github.com/appcelerator/amp/tests/cli   18.274s

[bquenin]

Yeah unfortunately, tests are also failing for me:

 bq@home  ~/go/src/github.com/appcelerator/amp   cli-test-platform-commands ± go test ./tests/cli
--- FAIL: TestCliCmds (0.15s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x480f0e]

goroutine 5 [running]:
panic(0x58ff80, 0xc42000a1a0)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
testing.tRunner.func1(0xc420076180)
        /usr/local/go/src/testing/testing.go:579 +0x25d
panic(0x58ff80, 0xc42000a1a0)
        /usr/local/go/src/runtime/panic.go:458 +0x243
github.com/appcelerator/amp/tests/cli.runCmdSpec(0xc420076180, 0x1, 0xc4200b6100, 0x11, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/bq/go/src/github.com/appcelerator/amp/tests/cli/cli_test.go:207 +0x95e
github.com/appcelerator/amp/tests/cli.runFrameworkSpec(0xc420076180, 0xc420014a50)
        /home/bq/go/src/github.com/appcelerator/amp/tests/cli/cli_test.go:136 +0x172
github.com/appcelerator/amp/tests/cli.runFramework(0xc420076180, 0xc420032098, 0x1, 0x1)
        /home/bq/go/src/github.com/appcelerator/amp/tests/cli/cli_test.go:114 +0x4e
github.com/appcelerator/amp/tests/cli.TestCliCmds(0xc420076180)
        /home/bq/go/src/github.com/appcelerator/amp/tests/cli/cli_test.go:90 +0x571
testing.tRunner(0xc420076180, 0x5d2da8)
        /usr/local/go/src/testing/testing.go:610 +0x81
created by testing.(*T).Run
        /usr/local/go/src/testing/testing.go:646 +0x2ec
FAIL    github.com/appcelerator/amp/tests/cli   0.150s

[bquenin]

Also, and maybe I'm a bit harsh, but I don't really think we need these tests because Travis already pull and start the platform.
It leaves us with amp pf status and amp pf stop.
Maybe you could add an amp pf status test in the CLI tests assuming the platform is already started (by travis)
We could also add an amp pf stop to .travis.yml and make sure it returns no error to automate the process.
Please everyone comment, I'd like to have your point of view on the matter.
Thanks!

[JosephGJ]

I've made a minor change. The tests were failing due to .Error() call in the mismatched expectation error return. In addition, I made the amp pf pull regex more forgiving as image versions are going to get updated repeatedly.

I like your idea @bquenin . Adding the amp pf status and amp pf stop to the travis build would be a good idea. The reason these commands were created was to dockerize the cli tests, and not just dockerize them but run them in docker-in-docker for complete isolation. However, due to some issues we had with running it, the idea was dropped.

I still think the setup and teardown directories for the cli_tests, along with the general restructuring of cli_test.go and the addition of the run field in the yaml files are useful additions to the cli tests, but the platform yaml files can definitely be removed if they aren't necessary.

[ndegory]

change request on test description format

[ndegory]

can we have a default value for run (true)? this way we only have to specify this key when we want to disable a test.

[JosephGJ]

@ndegory changed to a skip field, so the default value is false and you can give true to skip said command

[bquenin]

@JosephGJ Do you want me to remove the amp pf start and stop tests and only keep the amp pf statu test in the cli tests ?
Or do you want to do it ?

[JosephGJ]

@bquenin I've removed the amp pf start and stop commands, as well as any other unnecessary command tests.

[bquenin]

LGTM, tested on linux