These tests aren't actually new, they are old tests that had to be broken out to a new shard in https://github.com/flutter/flutter/pull/151433/. So they've already been passing for some time, and are still passing on their new shard, so they should be safe to mark as not `bringup:true`.
Reverts: flutter/flutter#151608
Initiated by: zanderso
Reason for reverting: Overeager cleanup of unused device spec.
Original PR Author: zanderso
Reviewed By: {christopherfujino, jonahwilliams}
This change reverts the following previous change:
Next step of https://github.com/flutter/flutter/issues/148085
This is a very common usage of ad so we want to make sure it's performant.
From the video, it scrolls quite smoothly, but we want to see the numbers, and keep monitoring it in case of regression.
https://github.com/flutter/flutter/assets/41930132/c7811c15-ac07-4989-a8a9-3c128e08cbe0
*List which issues are fixed by this PR. You must list at least one issue. An issue is not required if the PR fixes something trivial like a typo.*
Fixes https://github.com/flutter/flutter/issues/150230
*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
Reverts: flutter/flutter#150969
Initiated by: goderbauer
Reason for reverting: Failing test in https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8743574743030691569/+/u/run_android_obfuscate_test/stdout
Original PR Author: gmackall
Reviewed By: {christopherfujino, reidbaker}
This change reverts the following previous change:
After the land of https://github.com/flutter/engine/pull/53592, there is some log spam:
```
e: /Users/mackall/.gradle/caches/transforms-3/c1e137371ec1afe9bc9bd7b05823752d/transformed/fragment-1.7.1/jars/classes.jar!/META-INF/fragment_release.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.8.0, expected version is 1.6.0.
e: /Users/mackall/.gradle/caches/transforms-3/d86c7cb1c556fe1655fa56db671c649c/transformed/jetified-activity-1.8.1/jars/classes.jar!/META-INF/activity_release.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.8.0, expected version is 1.6.0.
...
```
I think this is harmless, but still annoying. Upgrading the AGP version fixes it. To be honest, I don't know why (I expected the Kotlin version would do it). But after https://github.com/flutter/flutter/pull/146307, our tests have been running on AGP/Gradle 8.1/8.3 for a while, so it makes sense to upgrade anyways.
In a follow up PR:
1. Also upgrade the tests that were left behind in https://github.com/flutter/flutter/pull/146307, as I think removal of discontinued plugins paved the way here.
After the land of https://github.com/flutter/engine/pull/53592, there is some log spam:
```
e: /Users/mackall/.gradle/caches/transforms-3/c1e137371ec1afe9bc9bd7b05823752d/transformed/fragment-1.7.1/jars/classes.jar!/META-INF/fragment_release.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.8.0, expected version is 1.6.0.
e: /Users/mackall/.gradle/caches/transforms-3/d86c7cb1c556fe1655fa56db671c649c/transformed/jetified-activity-1.8.1/jars/classes.jar!/META-INF/activity_release.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.8.0, expected version is 1.6.0.
...
```
I think this is harmless, but still annoying. Upgrading the AGP version fixes it. To be honest, I don't know why (I expected the Kotlin version would do it). But after https://github.com/flutter/flutter/pull/146307, our tests have been running on AGP/Gradle 8.1/8.3 for a while, so it makes sense to upgrade anyways.
In a follow up PR:
1. Also upgrade the tests that were left behind in https://github.com/flutter/flutter/pull/146307, as I think removal of discontinued plugins paved the way here.
This suite typically takes almost 30 minutes on CI, and some recent
engine->framework rolls failed because the tests exceeded the current 30
minute timeout.
On Linux hosts, there are currently 5 mokey's in staging and 7 in prod.
On Linux hosts, there is currently 1 MotoG4 in staging and 11 in prod.
This PR shifts the ~5 tests that have been running on mokey's for awhile
now from staging to prod, and shifts about half of the MotoG4 benchmarks
from MotoG4's in prod to mokey's in staging.
Following this change, we can disconnect half of the Linux/MotoG4's in
the prod pool and replace them with mokey's in the prod pool.
Part of https://github.com/flutter/flutter/issues/148085
We never made much use of the a02s's, so we can easily free up some
hosts for more mokey devices by eagerly shifting some of the tests on
a02s to mokey.
The test on MotoG4 has lots of flaky timeouts, so this PR moves it to
bringup to stop turning the tree red. This PR also adds the test on
Mokey in anticipation of removing the MotoG4 devices and to see if it
might be flaky there as well.
The macOS devicelab machines are not physically configured for benchmarking at a 60 Hz refresh rate. With Flutter vsync wired up via CVDisplayLink (rather than a hardcoded 60 Hz rate as previously), the bot shows 0 monitors configured and syncs at 30 Hz. This is not a meaningful target to measure against but consumes devicelab CPU cycles.
Given that there is currently no plan in place to acquire and configure a consistent pool of physical benchmarking desktop machines, disabling this test to reduce load on the bots.
See https://github.com/flutter/engine/pull/51210 for a discussion of the decision to move forward with wiring up vsync from macOS and ignore this benchmark.
For future archaeologists, I don't think it's worth removing all the plumbing in the Dart code (in [dev/devicelab/lib/tasks/perf_tests.dart](9d47055640/dev/devicelab/lib/tasks/perf_tests.dart (L963))) that supports running this benchmark on macOS. If we ever acquire hardware suitable for benchmarking with the appropriate HDMI dongle to simulate a 60Hz 4k display, for example, it would be reasonable to re-enable this benchmark.
Relands https://github.com/flutter/flutter/pull/146181.
Just 3 commits:
1. a revert of the revert
2. the fix described in https://github.com/flutter/flutter/pull/146181#issuecomment-2038238869
3. updating two postsubmit tests from Java 11 to 17, as that is required for this new AGP version.
I've verified that `flutter build apk --flavor paid --debug` fails in `dev/integration_tests/flavors/` with the error in ci without (2), and succeeds with it.
I've also verified that the `dev/benchmarks/complex_layout` app builds successfully with Java 17.
That covers all the postsubmits that failed according [to the dashboard](https://flutter-dashboard.appspot.com/#/build).
* Adds support for `flutter test --wasm`.
* The test compilation flow is a bit different now, so that it supports compilers other than DDC. Specifically, when we run a set of unit tests, we generate a "switchboard" main function that imports each unit test and runs the main function for a specific one based off of a value set by the JS bootstrapping code. This way, there is one compile step and the same compile output is invoked for each unit test file.
* Also, removes all references to `dart:html` from flutter/flutter.
* Adds CI steps for running the framework unit tests with dart2wasm+skwasm
* These steps are marked as `bringup: true`, so we don't know what kind of failures they will result in. Any failures they have will not block the tree at all yet while we're still in `bringup: true`. Once this PR is merged, I plan on looking at any failures and either fixing them or disabling them so we can get these CI steps running on presubmit.
This fixes https://github.com/flutter/flutter/issues/126692
Switch hot_mode_dev_cycle_linux__benchmark to run in postsubmit.
*List which issues are fixed by this PR. You must list at least one issue. An issue is not required if the PR fixes something trivial like a typo.*
Part of https://github.com/flutter/flutter/issues/145242
*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
This is required to match a rename being made for the devices in firebase lab.
Also removed `version=24` entry, since thats not supported on Nexus5x, per the firebase team.
Bug:b/322830672
*Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.*
*List which issues are fixed by this PR. You must list at least one issue. An issue is not required if the PR fixes something trivial like a typo.*
*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
Reverts: flutter/flutter#145224
Initiated by: hellohuanlin
Reason for reverting: breaks the tree
Original PR Author: hellohuanlin
Reviewed By: {gmackall, jmagman}
This change reverts the following previous change:
Reland https://github.com/flutter/flutter/pull/144745, which got reverted due to Android lockfile. Fixed by `dart dev/tools/bin/generate_gradle_lockfiles.dart`
*List which issues are fixed by this PR. You must list at least one issue. An issue is not required if the PR fixes something trivial like a typo.*
Fixes https://github.com/flutter/flutter/issues/143534
Fixes https://github.com/flutter/flutter/issues/143257
*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
These tests were failing in postsubmit after a change to `packages/flutter_tools`. We should run these in presubmit when `packages/flutter_tools` changes.
Windows Defender sometimes kills the `Windows_arm64 plugin_test_windows` test process, causing the test to hang until it times out after 30 minutes. This reduces the test timeout to 900 seconds (15 minutes) to recover from this scenario faster.
Test timeout duration was chosen by looking at successful duration percentiles in the last 100 days:
duration_seconds_p90 | duration_seconds_p99 | duration_seconds_max
-- | -- | --
532 | 545 | 576
BigQuery SQL:
```sql
WITH
successful_steps AS (
SELECT
b.id,
TIMESTAMP_DIFF(s.end_time, s.start_time, SECOND) AS duration_seconds,
FROM cr-buildbucket.flutter.builds AS b, UNNEST(steps) AS s
WHERE
create_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 100 DAY)
AND regexp_substr(input.gitiles_commit.project, '[^\\/]+$') = 'flutter'
AND builder.project || '/' || builder.bucket || '/' || builder.builder
= 'flutter/prod/Windows_arm64 plugin_test_windows'
AND name = 'run plugin_test_windows'
AND s.status = 'SUCCESS'
)
SELECT
percentiles[offset(90)] AS duration_seconds_p90,
percentiles[offset(99)] AS duration_seconds_p99,
duration_seconds_max
FROM (
SELECT
APPROX_QUANTILES(duration_seconds, 100) AS percentiles,
MAX(duration_minutes) AS duration_seconds_max
FROM successful_steps
);
```
`test_timeout_secs` is documented here: https://github.com/flutter/cocoon/blob/main/CI_YAML.md
Part of https://github.com/flutter/flutter/issues/145072
