"Fix failing checks" wiki page for new contributors (#154629)
Resolves https://github.com/flutter/flutter/issues/154628
<br>
As a contributor, I would expect resources related to testing to be located in the [/docs/contributing/testing](https://github.com/flutter/flutter/tree/master/docs/contributing/testing) directory. This PR adds a page to that directory geared toward helping new contributors.
<br>
I've noticed that in `team-framework` triage meetings, comments along the lines of "could you take a look at the failing checks?" are a regular occurrence.
I believe we could save quite a bit of effort by changing this:
> [**Understanding a LUCI build failure**](a5ca16ea94/docs/infra/Understanding-a-LUCI-build-failure.md)
to this:
> [**Flutter wiki â How to fix failing checks**](https://github.com/nate-thegrate/flutter/blob/fix-checks-wiki-page/docs/contributing/testing/Fix-failing-checks.md)
This commit is contained in:
Nate Wilson
GitHub
parent
ed553d10cf
commit
e77093bba4
138
docs/contributing/testing/Fix-failing-checks.md
Normal file
138
docs/contributing/testing/Fix-failing-checks.md
Normal file
@ -0,0 +1,138 @@
|
||||
|
||||
|
||||
# How to fix a PR's failing checks
|
||||
|
||||
<br>
|
||||
|
||||
### tree-status
|
||||
|
||||
|
||||
|
||||
Unlike other checks, **tree-status** isn't tied to the pull request:
|
||||
instead, it shows whether checks are passing in the main branch.\
|
||||
Failures can happen for a variety of reasons and should be addressed
|
||||
before anything else is merged in.
|
||||
|
||||
**What to do:** Once [review requirements](../Tree-hygiene.md#getting-a-code-review)
|
||||
are met and all other checks are passing, a reviewer will add the
|
||||
[**`autosubmit`**](../../infra/Landing-Changes-With-Autosubmit.md) label,
|
||||
and then a bot will merge the PR once **tree-status** succeeds.
|
||||
|
||||
<br>
|
||||
|
||||
### Google testing
|
||||
|
||||
|
||||
|
||||
A Google testing failure could be a flake ([see below](#flaking)), or it
|
||||
might be due to changes in the PR (See
|
||||
[Understanding Google Testing](../../infra/Understanding-Google-Testing.md)
|
||||
for more info).
|
||||
Google employees can view the test output and give feedback accordingly.
|
||||
|
||||
**What to do:** If 2 weeks have gone by and nobody's looked into it,
|
||||
feel free to [reach out on Discord](../Chat.md).
|
||||
|
||||
<br>
|
||||
|
||||
### ci.yaml validation
|
||||
|
||||
|
||||
|
||||
In order for checks to run correctly, the [.ci.yaml](../../../.ci.yaml)
|
||||
file needs to stay in sync with the base branch.
|
||||
|
||||
**What to do:** This check failure can be fixed by applying the latest changes
|
||||
from master.\
|
||||
(The [Tree hygiene](../Tree-hygiene.md#using-git) page recommends updating
|
||||
via rebase, rather than a merge commit.)
|
||||
|
||||
|
||||
|
||||
<br>
|
||||
|
||||
## A bug in the PR
|
||||
|
||||
Oftentimes, a change inadvertently breaks expected behavior.\
|
||||
When this happens, usually the best way to find out what's wrong is to
|
||||
[**view the test output**](#view-the-test-output).
|
||||
|
||||
If a **customer_testing** check is unsuccessful, it's a signal that something in the
|
||||
[Flutter customer test registry](https://github.com/flutter/tests/) has failed.
|
||||
This includes [package tests](../../ecosystem/testing/Understanding-Packages-tests.md)
|
||||
along with other tests from open-source Flutter projects.\
|
||||
If a pull request requires an update to those external tests, it qualifies as a
|
||||
[**breaking change**](../Tree-hygiene.md#handling-breaking-changes);
|
||||
please avoid those when possible.
|
||||
|
||||
If **Linux Analyze** fails, it's likely that one or more changes in the PR
|
||||
violated a [linter rule](https://dart.dev/lints/).\
|
||||
Consider reviewing the steps outlined in
|
||||
[setting up the framework dev environment](../../Setting-up-the-Framework-development-environment.md)
|
||||
so that most of these problems get caught in static analysis right away.
|
||||
|
||||
> [!NOTE]
|
||||
> All Dart code is run through static analysis:
|
||||
> this includes markdown code snippets in doc comments!
|
||||
>
|
||||
> See [Hixie's Natural Log](https://ln.hixie.ch/?start=1660174115) for more details.
|
||||
|
||||
<br>
|
||||
|
||||
### View the test output
|
||||
|
||||
Click on **Details** for the failing test, and then click
|
||||
**View more details on flutter-dashboard**.
|
||||
|
||||
|
||||
|
||||
The full test output is linked at the bottom of the page.
|
||||
|
||||
|
||||
|
||||
<br>
|
||||
|
||||
Often, there will be a message that resembles the one below:
|
||||
|
||||
```
|
||||
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
|
||||
The following TestFailure was thrown running a test:
|
||||
Expected: exactly one matching candidate
|
||||
Actual: _TextWidgetFinder:<Found 0 widgets with text
|
||||
"AsyncSnapshot<String>(ConnectionState.waiting, null, null, null)": []>
|
||||
Which: means none were found but one was expected
|
||||
|
||||
When the exception was thrown, this was the stack:
|
||||
#4 main.<anonymous closure>.<anonymous closure> (…/packages/flutter/test/widgets/async_test.dart:115:7)
|
||||
<asynchronous suspension>
|
||||
#5 testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:189:15)
|
||||
<asynchronous suspension>
|
||||
#6 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1032:5)
|
||||
<asynchronous suspension>
|
||||
<asynchronous suspension>
|
||||
(elided one frame from package:stack_trace)
|
||||
|
||||
This was caught by the test expectation on the following line:
|
||||
file:///b/s/w/ir/x/w/flutter/packages/flutter/test/widgets/async_test.dart line 115
|
||||
The test description was:
|
||||
gracefully handles transition from null future
|
||||
════════════════════════════════════════════════════════════════════════════════════════════════════
|
||||
```
|
||||
|
||||
From there, it's just a matter of finding the failing test,
|
||||
[running it locally](./Running-and-writing-tests.md),
|
||||
and figuring out how to fix it!
|
||||
|
||||
<br>
|
||||
|
||||
### Flaking
|
||||
|
||||
A check might "flake", or randomly fail, for a variety of reasons.
|
||||
|
||||
Sometimes a flake resolves itself after changes are pushed to re-trigger
|
||||
the checks. Consider [performing a rebase](#ciyaml-validation) to include
|
||||
the latest changes from the main branch.
|
||||
|
||||
Flakes often happen due to **infra errors**.
|
||||
For information on how to view and report infrastructure bugs, see the
|
||||
[infra failure overview](../../infra/Understanding-a-LUCI-build-failure.md#overview-of-an-infra-failure-build).
|
||||
@ -51,3 +51,5 @@ Please refer to the above example of the infra failure.
|
||||
4. Check if a flaky bug has been filed in [the flaky issues list](https://github.com/flutter/flutter/issues?q=is%3Aopen+is%3Aissue+label%3A%22c%3A+flake%22)
|
||||
5. File a new bug if needed
|
||||
6. If a rerun is needed, please refer to step 6 in the above infra failure session.
|
||||
|
||||
See also: [How to fix a PR's failing checks](../contributing/testing/Fix-failing-checks.md)
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user