This updates the codesigning test to account for iOS and macOS binaries in the artifact cache that are _expected_ to not be codesigned.
In https://github.com/flutter/engine/pull/54414 we started bundling dSYM (debugging symbols) within Flutter.xcframework, a requirement for App Store verification using Xcode 16.
We did the same for macOS in https://github.com/flutter/engine/pull/54696.
Unlike the framework dylib, dSYM contents are not directly codesigned (though the xcframework containing them is).
Issue: https://github.com/flutter/flutter/issues/154571
This was partially fixed in flutter/flutter#153787, which added just the extension-safe simulator dSYM, but we also bundle:
* arm64 release dSYM
* arm64 release extension-safe dSYM
* arm64_x86_64 simulator dSYM
In a followup engine patch, we'll stop bundling both the regular and extension-safe simulator dSYMs, but this gets the test passing at tip-of-tree.
Issue: https://github.com/flutter/flutter/issues/116493
A pending change to the flutter_drone LUCI recipe is using a logging library that sets the FLUTTER_TEST_OUTPUTS_DIR environment variable (see https://flutter.googlesource.com/recipes/+/b111cd3ef2297e66905cc48c2cfedce0cf1ba46f)
flutter_driver uses FLUTTER_TEST_OUTPUTS_DIR as its output directory if it is set. run_web_tests invokes flutter_driver and needs to look for outputs in that directory.
When attempting to release the Flutter 3.24.1 hotfix, Flutter framework post submits failed due to the following error:
```
âââ¡ERROR #1âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ
â UNEXPECTED ERROR!
â Exception: Found unexpected binary in cache: /Volumes/Work/s/w/ir/x/w/flutter/bin/cache/artifacts/engine/ios-release/extension_safe/Flutter.xcframework/ios-arm64_x86_64-simulator/dSYMs/Flutter.framework.dSYM/Contents/Resources/DWARF/Flutter
â #0 verifyExist (file:///Volumes/Work/s/w/ir/x/w/flutter/dev/bots/suite_runners/run_verify_binaries_codesigned_tests.dart:150:12)
â <asynchronous suspension>
â #1 verifyCodesignedTestRunner (file:///Volumes/Work/s/w/ir/x/w/flutter/dev/bots/suite_runners/run_verify_binaries_codesigned_tests.dart:28:3)
â <asynchronous suspension>
â #2 _runFromList (file:///Volumes/Work/s/w/ir/x/w/flutter/dev/bots/utils.dart:601:5)
â <asynchronous suspension>
â #3 main (file:///Volumes/Work/s/w/ir/x/w/flutter/dev/bots/test.dart:125:5)
â <asynchronous suspension>
â
â The test.dart script should be corrected to catch this error and call foundError().
â Some tests are likely to have been skipped.
ââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ
```
I've updated the tests to include the listed artifact.
### Open questions
1. Is it possible that the test failed only on the first binary?
2. Do we expect a need to add the remainder of the [entitlement changes](c9b9d5780d)?
3. I am under the assumption that the bots pull from master and are not branch specific, is this a correct assumption?
CC @christopherfujino @cbracken
Closes https://github.com/flutter/flutter/issues/152325.
This PR is large due to generate `flutter create --platforms android`. A quick summary:
- Moves the integration test from `packages/flutter_driver/test` to `dev/integration_tests`
- Created a sample Flutter app that draws a blue rectangle
- Forked a subset of `package:flutter_goldens` that will work on the standalone Dart VM
- Forked a subset of `goldens.dart` (from `flutter_test`) to `src/native/goldens.dart` (i.e. `matchesGoldenFile`)
This ... works locally, but as usual I have no idea if it will work on Skia Gold so let's roll some dice.
Turns out just supporting the right value for `kDebugMode` was a lot simpler than I thought. Debug builds used to never go through the build system code path when using `flutter run`, but now that we have wasm this can occur with the run command.
This should address https://github.com/flutter/flutter/issues/148850
Reverts: flutter/flutter#150969
Initiated by: goderbauer
Reason for reverting: Failing test in https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8743574743030691569/+/u/run_android_obfuscate_test/stdout
Original PR Author: gmackall
Reviewed By: {christopherfujino, reidbaker}
This change reverts the following previous change:
After the land of https://github.com/flutter/engine/pull/53592, there is some log spam:
```
e: /Users/mackall/.gradle/caches/transforms-3/c1e137371ec1afe9bc9bd7b05823752d/transformed/fragment-1.7.1/jars/classes.jar!/META-INF/fragment_release.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.8.0, expected version is 1.6.0.
e: /Users/mackall/.gradle/caches/transforms-3/d86c7cb1c556fe1655fa56db671c649c/transformed/jetified-activity-1.8.1/jars/classes.jar!/META-INF/activity_release.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.8.0, expected version is 1.6.0.
...
```
I think this is harmless, but still annoying. Upgrading the AGP version fixes it. To be honest, I don't know why (I expected the Kotlin version would do it). But after https://github.com/flutter/flutter/pull/146307, our tests have been running on AGP/Gradle 8.1/8.3 for a while, so it makes sense to upgrade anyways.
In a follow up PR:
1. Also upgrade the tests that were left behind in https://github.com/flutter/flutter/pull/146307, as I think removal of discontinued plugins paved the way here.
After the land of https://github.com/flutter/engine/pull/53592, there is some log spam:
```
e: /Users/mackall/.gradle/caches/transforms-3/c1e137371ec1afe9bc9bd7b05823752d/transformed/fragment-1.7.1/jars/classes.jar!/META-INF/fragment_release.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.8.0, expected version is 1.6.0.
e: /Users/mackall/.gradle/caches/transforms-3/d86c7cb1c556fe1655fa56db671c649c/transformed/jetified-activity-1.8.1/jars/classes.jar!/META-INF/activity_release.kotlin_module: Module was compiled with an incompatible version of Kotlin. The binary version of its metadata is 1.8.0, expected version is 1.6.0.
...
```
I think this is harmless, but still annoying. Upgrading the AGP version fixes it. To be honest, I don't know why (I expected the Kotlin version would do it). But after https://github.com/flutter/flutter/pull/146307, our tests have been running on AGP/Gradle 8.1/8.3 for a while, so it makes sense to upgrade anyways.
In a follow up PR:
1. Also upgrade the tests that were left behind in https://github.com/flutter/flutter/pull/146307, as I think removal of discontinued plugins paved the way here.
The previous expression `success || line.contains('--- TEST FAILED ---')` made it such that as soon as success is reported, the test would send "q" on every line of stdout, which is wrong. This fixes the condition such that "q" is sent once, only when `--- TEST SUCCEEDED ---` or `--- TEST FAILED ---` is printed.
Refactor the framework + test harness test suites in order to reduce testing logic in test.dart and allow for later implementing package:test onto the existing tests
The refactor of both suites included in this PR because they both depended on util functions and variables that also needed to be refactored.
Part of https://github.com/flutter/flutter/issues/145482
Refactor framework coverage tests in order to reduce testing logic in test.dart and allow for later implementing package:test onto the existing framework coverage tests
Part of https://github.com/flutter/flutter/issues/145482
Move from `test.dart` into new `runner_utils.dart` file:
- shard calculation related methods + `shuffleSeed` logic
- env variable flag calculation
- runner wrapper methods `runDartTest` and `_runFromList`
Create web tests suite class containing:
- web long running tests
- runWebHtmlUnitTests
- runWebCanvasKitUnitTests
- runWebSkwasmUnitTests
- the associate helper methods with the above
This pull request aims for improved readability, based on issue #146600.
```dart
// before
Set<Color> _distinctVisibleColors() {
final Set<Color> distinctVisibleColors = <Color>{};
if (top.style != BorderStyle.none) {
distinctVisibleColors.add(top.color);
}
if (right.style != BorderStyle.none) {
distinctVisibleColors.add(right.color);
}
if (bottom.style != BorderStyle.none) {
distinctVisibleColors.add(bottom.color);
}
if (left.style != BorderStyle.none) {
distinctVisibleColors.add(left.color);
}
return distinctVisibleColors;
}
// after
Set<Color> _distinctVisibleColors() {
return <Color>{
if (top.style != BorderStyle.none) top.color,
if (right.style != BorderStyle.none) right.color,
if (bottom.style != BorderStyle.none) bottom.color,
if (left.style != BorderStyle.none) left.color,
};
}
```
Most of the repo should be covered in this PR (aside from `flutter_tools/`, since there was a lot going on in there).
This adds support for adding the `--wasm` flag to `flutter run` and `flutter drive`
* Emits errors if you attempt to use the skwasm renderer without the `--wasm` flag
* Emits errors if you try to use `--wasm` when not using a web device
* Uses the skwasm renderer by default if you pass `--wasm` and no `--web-renderer`
Refactor verify codesigned tests in order to reduce testing logic in test.dart and allow for later implementing package:test onto the existing verify codesigned tests
Part of https://github.com/flutter/flutter/issues/145482
Refactor fuchsia_precache suite in order to reduce testing logic in test.dart and allow for later implementing package:test onto the existing fuchsia_precache tests
Part of https://github.com/flutter/flutter/issues/145482
Refactor customer_testing suite in order to reduce testing logic in test.dart and allow for later implementing package:test onto the existing customer_testing tests
Part of https://github.com/flutter/flutter/issues/145482
*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
Refactor docs testing suite in order to reduce testing logic in test.dart and allow for later implementing package:test onto the existing docs tests
Part of https://github.com/flutter/flutter/issues/145482
Refactor the analyze testing suite in order to reduce testing logic in test.dart and allow for later implementing package:test onto the existing analyze tests
Part of https://github.com/flutter/flutter/issues/145482
Refactor realm_checker suite in order to reduce testing logic in test.dart and allow for later implementing package:test onto the existing realm_checker tests
Part of https://github.com/flutter/flutter/issues/145482
*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
Refactor flutter_plugins suite in order to reduce testing logic in test.dart and allow for later implementing package:test onto the existing flutter_plugins tests
Also update the test_test invocation of getFlutterPackagesVersion
Part of https://github.com/flutter/flutter/issues/145482
Refactor skp_generator_tests suite in order to reduce testing logic in test.dart and allow for later implementing package:test onto the existing skp_generator_tests tests
Part of https://github.com/flutter/flutter/issues/145482
Refactor web long running tests in order to reduce testing logic in test.dart, create a suite_runners directory and allow for later implementing package:test onto the existing tests
Part of https://github.com/flutter/flutter/issues/145482
Refactor add_to_app_life_cycle_tests in order to reduce testing logic in test.dart, create a suite_runners directory and allow for later implementing package:test onto add_to_app_life_cycle_tests
Part of https://github.com/flutter/flutter/issues/145482
