What would the pros / cons of using platform to run integration / functional tests, instead of a typical Continuous Integration service? And is using platform to run functional tests possible / a bad idea?
My thinking is this:
- Unit tests, static analysis, code linting etc. are easy to run in isolation, so that should remain in a typical CI environment
- Functional tests require a functioning environment, which platform.sh can automatically provide
- Sometimes you want to run your functional tests against production data, which is also something platform.sh can automatically provide
An example implementation could be:
- On build, detect if this is an environment you want to run tests in (eg. non-production, pull request environment etc.)
- Install any tools you need to run functional tests (eg. a headless browser and testing framework)
- Run the tests – fail the build if the tests fail
- Send notification of the passed/failed build (I think the standard github pull request integration would report this back, but I’m not sure – an email or slack notification could be sent)
The build environment is branch-agnostic. Since build images can be reused on multiple branches there’s no way to detect the branch during build.
This is not something we currently support directly; you can run unit tests (that do not require access to the database) during build. More on that here: https://community.platform.sh/t/how-to-fail-a-build-on-failing-unit-tests/57
For end to end tests, you would need to ensure the tools are installed in all environments. Then in a
post_deploy hook, check if you’re on a branch where you want to run tests (at minimum, non-
master), and run whatever commands you need. It’s better to do in
deploy blocks the environment being publicly accessible, which may be necessary depending on the nature of your tests.
The report-back in case of failure is the big challenge. At present we have no mechanism available to do that. If you have some external notification system that can accept a webhook, you can have your test code ping that URL when it’s done. We have no built-in support for it at this time, though.