firka/flutter
firka/flutter
5
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[devicelab] Refresh documentation (#80064)

Browse Source
This commit is contained in:
Casey Hillers 2021-04-14 13:49:02 -07:00 committed by GitHub
parent d6fbfdb9cd
commit d55b8c0ada
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 45 additions and 56 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

101
dev/devicelab/README.md
View File

@ -7,23 +7,27 @@ the tests are referred to as "tasks" in the API, but since we primarily use it
for testing, this document refers to them as "tests".
Current statuses for the devicelab are available at
https://flutter-dashboard.appspot.com. See [dashboard user guide](https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md)
https://flutter-dashboard.appspot.com/#/build. See [dashboard user guide](https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md)
for information on using the dashboards.
## How the DeviceLab runs tasks
## Table of Contents
* [How the DeviceLab runs tests](#how-the-devicelab-runs-tests)
* [Running tests locally](#running-tests-locally)
* [Writing tests](#writing-tests)
* [Adding tests to continuous integration](#adding-tests-to-continuous-integration)
* [Adding tests to presubmit](#adding-tests-to-presubmit)
The DeviceLab devices continuously ask Flutter's continuous integration system
[Cocoon](https://github.com/flutter/cocoon) for tasks to run. When Cocoon has a
task that is suitable for the device (e.g. Android test), it reserves that
task for the device. See [manifest.yaml](manifest.yaml) for more information on
the information used for scheduling tasks.
1. If the task succeeds, the test runner reports the success to Cocoon. The dashboards
will show that task in green.
2. If the task fails, the test runner reports the failure to the server. Cocoon
increments the run attempt counter and puts the task back in the pool of available
tasks. If a task does not succeed after a certain number of attempts (as of this writing the limit is 2),
the task is marked as failed and is displayed using a red color on the dashboard.
## How the DeviceLab runs tests
DeviceLab tests are run against physical devices in Flutter's lab (the "DeviceLab").
Tasks specify the type of device they are to run on (`linux_android`, `mac_ios`, `mac_android`, `windows_android`, etc).
When a device in the lab is free, it will pickup tasks that need to be completed.
1. If the task succeeds, the test runner reports the success and uploads its performance metrics to Flutter's infrastructure. Not
all tasks record performance metrics.
2. If the task fails, the test runner reports the failure to Flutter's infrastructure and no performance metrics are collected
## Running tests locally
@ -46,30 +50,18 @@ Running the devicelab will do things to your environment.
Notably, it will start and stop Gradle, for instance.
### Running all tests
To run all tests defined in `manifest.yaml`, use option `-a` (`--all`):
```sh
../../bin/cache/dart-sdk/bin/dart bin/run.dart -a
```
This defaults to only running tests supported by your host device's platform
(`--match-host-platform`) and exiting after the first failure (`--exit`).
### Running specific tests
To run a test, use option `-t` (`--task`):
```sh
# from the .../flutter/dev/devicelab directory
../../bin/cache/dart-sdk/bin/dart bin/run.dart -t {NAME_OR_PATH_OF_TEST}
../../bin/cache/dart-sdk/bin/dart bin/test_runner.dart test -t {NAME_OR_PATH_OF_TEST}
```
Where `NAME_OR_PATH_OF_TEST` can be either of:
- the _name_ of a task, which you can find in the `manifest.yaml` file in this
directory. Example: `complex_layout__start_up`.
- the _name_ of a task, which is a file's basename in `bin/tasks`. Example: `complex_layout__start_up`.
- the path to a Dart _file_ corresponding to a task, which resides in `bin/tasks`.
Tip: most shells support path auto-completion using the Tab key. Example:
`bin/tasks/complex_layout__start_up.dart`.
@ -80,14 +72,6 @@ To run multiple tests, repeat option `-t` (`--task`) multiple times:
../../bin/cache/dart-sdk/bin/dart bin/run.dart -t test1 -t test2 -t test3
```
To run tests from a specific stage, use option `-s` (`--stage`).
Currently, there are only three stages defined, `devicelab`,
`devicelab_ios` and `devicelab_win`.
```sh
../../bin/cache/dart-sdk/bin/dart bin/run.dart -s {NAME_OF_STAGE}
```
### Running tests against a local engine build
To run device lab tests against a local engine build, pass the appropriate
@ -211,31 +195,36 @@ A task runs in its own standalone Dart VM and reports results via Dart VM
service protocol. This ensures that tasks do not interfere with each other and
lets the CI system time out and clean up tasks that get stuck.
## Adding tests to the CI environment
## Adding tests to continuous integration
The `manifest.yaml` file describes a subset of tests we run in the CI. To add
your test edit `manifest.yaml` and add the following in the "tasks" dictionary:
Host only tests should be added to `flutter_tools`.
```
{NAME_OF_TEST}:
description: {DESCRIPTION}
stage: {STAGE}
required_agent_capabilities: {CAPABILITIES}
```
There are several PRs needed to add a DeviceLab task to CI.
Where:
- `{NAME_OF_TEST}` is the name of your test that also matches the name of the
_TASK_- the name of your test that also matches the name of the
file in `bin/tasks` without the `.dart` extension.
- `{DESCRIPTION}` is the plain English description of your test that helps
others understand what this test is testing.
- `{STAGE}` is `devicelab` if you want to run on Android, or `devicelab_ios` if
you want to run on iOS.
- `{CAPABILITIES}` is an array that lists the capabilities required of
the test agent (the computer that runs the test) to run your test. As of writing,
the available capabilities are: `linux`, `linux/android`, `linux-vm`,
`mac`, `mac/ios`, `mac/iphonexs`, `mac/ios32`, `mac-catalina/ios`,
`mac-catalina/android`, `ios/gl-render-image`, `windows`, `windows/android`.
1. Add prod builder to [flutter/infra devicelab_config.star](https://github.com/flutter/infra/blob/master/config/devicelab_config.star)
- Example PR: https://github.com/flutter/infra/pull/401/files
- This will need to soak for 15 minutes after merged to propagate (should show up in [LUCI console[(https://ci.chromium.org/p/flutter/g/devicelab/console))
- There are various lists for the different testbeds a test can run on
2. Add task to [flutter/flutter prod_builders.json](https://github.com/flutter/flutter/blob/master/dev/prod_builders.json)
- Example PR: https://github.com/flutter/flutter/pull/79913/files
- Set `flaky: true` for validation before blocking tree
3. After 10 green runs, update [flutter/flutter prod_builders.json](https://github.com/flutter/flutter/blob/master/dev/prod_builders.json) to `flaky:false`
If your test needs to run on multiple operating systems, create a separate test
for each operating system.
## Adding tests to presubmit
Flutter's DeviceLab does not currently have capacity to run tests against physical devices in presubmit.
Note that DeviceLab tests should generally require a tethered device. If you are adding host-only tests, considering adding your test to `packages/flutter_tools/test/integration.shard`. Example: https://github.com/flutter/flutter/pull/73577/files"
1. Add try builder to [flutter/infra devicelab_config.star](https://github.com/flutter/infra/blob/master/config/devicelab_config.star)
- Example PR: https://github.com/flutter/infra/pull/401/files
- This will need to soak for 15 minutes after merged to propagate
- There are various lists for the different testbeds a test can run on
2. Add task to [flutter/flutter try_builders.json](https://github.com/flutter/flutter/blob/master/dev/try_builders.json)
- Example PR: https://github.com/flutter/flutter/pull/79913/files