flutter/docs/infra/Understanding-a-LUCI-build-failure.md
Kate Lovett 6832ac7a95
Update docs after #165258 (#165716)
Follow up from #165258 
Removes references to the deleted Infra-Ticket-Queue.md, and mentions of
the Infra Ticket Queue Github project.
Not sure if we should keep the issue template?

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [ ] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
2025-03-25 21:09:57 +00:00

4.5 KiB

Most tests have been running in Flutter CI LUCI for multiple flutter repositories, including post-submit framework, pre-submit framework, etc. This page uses framework as an example, and talks about what to do when a LUCI build failure happens.

Build dashboards

Infra Failure

An infra failure comes with network connection issues, hardware outage, recipe breakage, cipd dependency issues, etc. It shows up as a purple box in the dashboards:

  • Framework post-submit build dashboard: three green squares, a purple square, and four green squares, representing build results
  • Framework pre-submit build console: one green rectangle, a purple rectangle, and one more green rectangle, representing build results

Overview of an infra failure build

An example build: Linux color_filter_and_fade_perf__e2e_summary

  • (i) Link to the historical build list of this builder
  • (ii) A quick glimpse of the infra failure
  • (iii) The step list of this builder, defined by the recipe on the right
  • (iv) The real failed step causing the build failure
  • (v) Check stdout for detailed log

What to do

  1. Check if the infra failure has happened on earlier builds by clicking (i)
  2. Check if issue already exists in the infra bug pool
  3. If not, file an infra bug
  4. If you want to get an immediate help, please ask in the discord hackers-infra channel
  5. If this is an infra flake, and a retry is needed
    • For pre-submit test, click Re-run in the check run page. The presubmit rerun interface
      • Limited to flutter-hackers group.
      • Ask a team member to re-run in Chat channel #hackers-infra if you don't have access.
    • For post-submit test, login to framework build dashboard, click the task box, and click RERUN. The post submit rerun interface
      • Limited to Googlers currently due to some technical limitations of our infrastructure.
      • Ask a Googler to re-run in Chat channel #hackers-infra.

Test Failure

A test failure shows up as a red box in the dashboards:

  • Framework post-submit build dashboard: two green squares, a red square, and three green squares, representing build results
  • Framework pre-submit build console: two green squares, a red square, and two green squares, representing build results

Overview of a test failure build

Please refer to the above example of the infra failure.

What to do

  1. Check if it happens in earlier builds/commits via (i)
  2. Debug based on the error message (ii) and detailed log (v) to see if a real test failure caused by code changes.
  3. Check if the issue already exists in the issues list
  4. Check if a flaky bug has been filed in the flaky issues list
  5. File a new bug if needed
  6. If a rerun is needed, please refer to step 6 in the above infra failure session.

See also: How to fix a PR's failing checks