What would the pros / cons of using platform to run integration / functional tests, instead of a typical Continuous Integration service? And is using platform to run functional tests possible / a bad idea?
My thinking is this:
Unit tests, static analysis, code linting etc. are easy to run in isolation, so that should remain in a typical CI environment
Functional tests require a functioning environment, which platform.sh can automatically provide
Sometimes you want to run your functional tests against production data, which is also something platform.sh can automatically provide
An example implementation could be:
On build, detect if this is an environment you want to run tests in (eg. non-production, pull request environment etc.)
Install any tools you need to run functional tests (eg. a headless browser and testing framework)
Run the tests – fail the build if the tests fail
Send notification of the passed/failed build (I think the standard github pull request integration would report this back, but I’m not sure – an email or slack notification could be sent)
For end to end tests, you would need to ensure the tools are installed in all environments. Then in a post_deploy hook, check if you’re on a branch where you want to run tests (at minimum, non-master), and run whatever commands you need. It’s better to do in post_deploy than deploy because deploy blocks the environment being publicly accessible, which may be necessary depending on the nature of your tests.
The report-back in case of failure is the big challenge. At present we have no mechanism available to do that. If you have some external notification system that can accept a webhook, you can have your test code ping that URL when it’s done. We have no built-in support for it at this time, though.