150 Commits

Author SHA1 Message Date
auto-submit[bot]
1cf57f1eaa
Reverts "Remove unused Cocoon components from devicelab/bin/test_runner.dart. (#165628)" (#165709)
<!-- start_original_pr_link -->
Reverts: flutter/flutter#165628
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: jtmcdole
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: breaking tree:


https://ci.chromium.org/ui/p/flutter/builders/prod/Linux_mokey%20backdrop_filter_perf__e2e_summary/3833/overview
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: matanlurey
<!-- end_original_pr_author -->

<!-- start_reviewers -->
Reviewed By: {jtmcdole, yjbanov}
<!-- end_reviewers -->

<!-- start_revert_body -->
This change reverts the following previous change:
🚫 **Do not merge** until
https://flutter-review.googlesource.com/c/recipes/+/64220/1 is merged.

---

Closes https://github.com/flutter/flutter/issues/165618.

The `devicelab/bin/test_runner.dart upload-metrics` command use to have
_two_ responsibilities:

- Well, upload test **metrics** (benchmarks) to Skia Perf (it still does
that)
- Upload test **status** to Cocoon (it did until
https://github.com/flutter/flutter/pull/165614)

As https://github.com/flutter/flutter/pull/165614 proved, this API
predated the current LUCI setup, where Cocoon itself receives task
status updates from LUCI, and it turns out this entire time, DeviceLab
was making (at best) NOP calls, and at worst, causing crashes and
corrupt data (https://github.com/flutter/flutter/issues/165610).

In other words, this is removing entirely dead/unused code (though the
recipes have to be updated first).

/cc @jason-simmons as I need Jason's help reviewing the recipes change
and want to provide context.
<!-- end_revert_body -->

Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
2025-03-21 22:30:47 +00:00
Matan Lurey
b06c05c6f4
Remove unused Cocoon components from devicelab/bin/test_runner.dart. (#165628)
🚫 **Do not merge** until
https://flutter-review.googlesource.com/c/recipes/+/64220/1 is merged.

---

Closes https://github.com/flutter/flutter/issues/165618.

The `devicelab/bin/test_runner.dart upload-metrics` command use to have
_two_ responsibilities:

- Well, upload test **metrics** (benchmarks) to Skia Perf (it still does
that)
- Upload test **status** to Cocoon (it did until
https://github.com/flutter/flutter/pull/165614)

As https://github.com/flutter/flutter/pull/165614 proved, this API
predated the current LUCI setup, where Cocoon itself receives task
status updates from LUCI, and it turns out this entire time, DeviceLab
was making (at best) NOP calls, and at worst, causing crashes and
corrupt data (https://github.com/flutter/flutter/issues/165610).

In other words, this is removing entirely dead/unused code (though the
recipes have to be updated first).

/cc @jason-simmons as I need Jason's help reviewing the recipes change
and want to provide context.
2025-03-21 21:27:48 +00:00
Michael Goderbauer
5491c8c146
Auto-format Framework (#160545)
This auto-formats all *.dart files in the repository outside of the
`engine` subdirectory and enforces that these files stay formatted with
a presubmit check.

**Reviewers:** Please carefully review all the commits except for the
one titled "formatted". The "formatted" commit was auto-generated by
running `dev/tools/format.sh -a -f`. The other commits were hand-crafted
to prepare the repo for the formatting change. I recommend reviewing the
commits one-by-one via the "Commits" tab and avoiding Github's "Files
changed" tab as it will likely slow down your browser because of the
size of this PR.

---------

Co-authored-by: Kate Lovett <katelovett@google.com>
Co-authored-by: LongCatIsLooong <31859944+LongCatIsLooong@users.noreply.github.com>
2024-12-19 20:06:21 +00:00
Kate Lovett
7b0b03bc0d
Adjust timeout for devicelab tests (#156553)
Fixes https://github.com/flutter/flutter/issues/156456

There is a known long running test that has been bumping up against the default timeout. This adjusts the perTestTimeout for the devicelab tests to add 15 seconds to stop this from flaking.

The test is question: 82ebb74c64/dev/devicelab/test/run_test.dart (L76)

`test/run_test.dart: run.dart script prints a message after a few seconds when failing to connect (this test takes >10s)`
2024-10-15 20:47:03 +00:00
Ben Konyi
2c84e63ba7
[ Cocoon ] Wait for task results to be received by the task runner before shutting down the task process (#156002)
Prior to this fix, `_TaskRunner.run` would immediately cleanup the
keep-alive port once the task completed, which would result in the
isolate shutting down as soon as the task result was returned from
`ext.cocoonRunTask` callback in the form of a
`ServiceExtensionResponse`. Since the service extension response is
processed by the service isolate, it was possible for the VM to start
shutting down before the service isolate could send the task result data
back to the task runner.

This change introduces a new service extension,
`ext.cocoonTaskResultReceived`, that the task runner invokes after it
receives the task result from `ext.cocoonRunTask`, notifying the task
process that it can close the keep-alive port and shutdown.

Fixes https://github.com/flutter/flutter/issues/155475
2024-10-02 17:01:24 -04:00
Matan Lurey
f070ffc53b
Get reasonable output when a stream test fails. (#154377)
Debugging https://github.com/flutter/flutter/issues/154268 was hard; when I removed `--disable-dart-dev` I would get the following error from this test:

<img width="627" alt="Screenshot 2024-08-29 at 4 25 43 PM" src="https://github.com/user-attachments/assets/2cc3266e-cc6e-425f-b909-12d7556ff110">

After my change, this is what the error looks like:

<img width="1055" alt="Screenshot 2024-08-29 at 4 26 50 PM" src="https://github.com/user-attachments/assets/8b0ecccc-a7c9-4da7-bf22-15cef24c4cc7">

In general we should embrace more of the test package and matchers when appropriate.
2024-09-03 21:36:24 +00:00
Jason Simmons
203a19e82f
Ensure that the output path used by perf_tests_test matches the path used by perf_tests if the FLUTTER_TEST_OUTPUTS_DIR variable is set (#153888)
A recent change to the flutter_drone recipe is setting FLUTTER_TEST_OUTPUTS_DIR in the environment (see https://flutter.googlesource.com/recipes/+/b111cd3ef2297e66905cc48c2cfedce0cf1ba46f)

If FLUTTER_TEST_OUTPUTS_DIR is set, then perf_tests will use an output path based on that variable.  perf_tests_test needs to use the same logic when looking for the expected output file.
2024-08-22 03:30:24 +00:00
Reid Baker
1eb7cd2c73
allow adb to set canfail then use canFail=true for clearing logs (#150517)
Fixes https://github.com/flutter/flutter/issues/150093

New tests added to cover that we at least pass the arguments we expect to adb. 

The test for #150093  is not ideal in that it does not verify the behavior of a failed process but instead ensures we set the parameter that contains the behavior we want. 

devicelab code and tests are not setup to enable fake process or fake output from stdin/stderr and hang if adb or no hardware are present.
2024-06-24 19:13:24 +00:00
Ian Hickson
5e216d44ca
Simplify devicelab logic and fix tests (#139122)
- fix https://github.com/flutter/flutter/issues/53707 by having the test not expect a timeout but instead actually look for the retry message
- simplify the `--task` option to only accept task names rather than also accepting paths
- remove some obsolete options that referred to the manifest which no longer seems to exist
2023-11-29 08:41:27 +00:00
Callum Moffat
2925525e35
Upload frame_request_pending_latency [attempt #2] (#135890)
These values are generated since https://github.com/flutter/flutter/pull/135279, but I didn't know to add the new keys to this list to get them to upload. 

Failed to do so in #135645, I believe the mistake there was putting them in `_kCommonScoreKeys`, which is also used in "E2E" tests, that don't get full trace data, only high level `FrameTiming` packets.

Part of https://github.com/flutter/flutter/issues/129150
2023-10-05 21:54:27 +00:00
Zachary Anderson
63b98d55b0
Revert "Upload generated frame-request-pending stats" (#135672)
Reverts flutter/flutter#135645
2023-09-28 07:23:25 -07:00
Callum Moffat
dd341d4295
Upload generated frame-request-pending stats (#135645)
These values are generated since https://github.com/flutter/flutter/pull/135279, but I didn't know to add the new keys to this list to get them to upload.

Part of https://github.com/flutter/flutter/issues/129150
2023-09-28 13:05:10 +00:00
Matan Lurey
a7997f606e
Update dev/devicelab/** to provide --local-engine-host. (#132342)
Partial work towards https://github.com/flutter/flutter/issues/132245.

I have to admit I don't totally understand what I've updated, or whether
there are more integration points needed.
2023-08-10 17:25:30 -07:00
Zachary Anderson
aef7929f3d
Report total_ui_gc_time to SkiaPerf (#125919)
Not sure why this wasn't already being reported.
2023-05-02 22:09:33 -07:00
Ben Konyi
ecd7518df5
Reland "Remove references to Observatory (#118577)" (#121606)
This reverts commit 275ab9c69bc15113003b77637ea90031df8ed348.
2023-02-28 11:57:04 -05:00
Michael Goderbauer
275ab9c69b
Revert "Reland "Remove references to Observatory (#118577)" (#121215)" (#121555)
Revert "Reland "Remove references to Observatory (#118577)""
2023-02-27 23:46:53 +00:00
Ben Konyi
fbae472fc3
Reland "Remove references to Observatory (#118577)" (#121215)
This reverts commit 298d8c76ba78007deb5b96f320a11ccefe97a794.
2023-02-27 09:26:43 -05:00
Casey Hillers
298d8c76ba
Revert "Remove references to Observatory (#118577)" (#120929)
This reverts commit 2df140f40d8c771ecef4f21cbdf635ec1235b4ac.
2023-02-16 21:28:30 -08:00
Ben Konyi
2df140f40d
Remove references to Observatory (#118577)
Observatory is being deprecated for Dart 3.0 so it should no longer be
referenced in tooling messaging / flags.

See https://github.com/dart-lang/sdk/issues/50233
2023-02-13 14:29:30 -05:00
Michael Thomsen
253e3c0856
Migrate device lab runner to nulls safety (#110582) 2022-08-30 19:46:05 +02:00
keyonghan
8027842393
Copy artifacts to applicationBinaryPath when specified for build+test separation (#109879) 2022-08-23 16:32:39 +00:00
Michael Goderbauer
10a7c9ba22
Unify analysis options (#108462) 2022-07-28 09:07:49 -07:00
Alexandre Ardhuin
93a74ca9f2
remove unused members in executable libs (#107464) 2022-07-12 21:41:08 +00:00
Alexandre Ardhuin
07f1c20474
add missing trailing commas in list/set/map literals (#102585) 2022-04-27 09:15:35 +02:00
keyonghan
647044c969
Use separate metric file names for same tests running on different platforms (#101317)
* use separate file names for same test

* use string buffer
2022-04-05 09:16:04 -07:00
Michael Goderbauer
195a1cc413
Prepare dev (minus flutter_gallery) for use_super_parameters (#100516) 2022-03-29 12:52:42 -07:00
Chris Yang
e6bd208196
Reland "Add the refresh rate fields to perf_test #99710" (#99854) 2022-03-21 15:55:13 -07:00
David Iglesias
93c0c043cf
[devicelab] Web benchmarks now run on Chromium 89+ (#98629) 2022-02-16 19:04:14 -08:00
keyonghan
e8bc5c5ace
Support Samsung phone when getting device wakefulness status (#98146) 2022-02-11 12:05:18 -08:00
Michael Goderbauer
5684b9f69e
Enable no_leading_underscores_for_local_identifiers (#96422) 2022-01-21 14:43:59 -08:00
Ian Hickson
57f097b532
Refactor devicelab bin/run.dart and other cleanup (#96320) 2022-01-10 13:40:17 -08:00
keyonghan
33be68d9bf
Stop sending metrics to builder name (#92989) 2021-11-05 16:28:02 -07:00
keyonghan
30d3866ab9
Add extra benchmark metrics to test name in addition to builder name (#92530) 2021-10-27 15:33:06 -07:00
keyonghan
7de92e2653
Add devicelab benchmark tags support (#92141) 2021-10-20 15:03:06 -07:00
Ian Hickson
61a0add286
Enable avoid_redundant_argument_values lint (#91409) (#91462) 2021-10-08 09:25:14 -07:00
Zachary Anderson
b9d2177da0
Revert "Enable avoid_redundant_argument_values lint (#91409)" (#91461)
This reverts commit 5fd259be244244e14ff87fee9b6a721625f952cd.
2021-10-07 21:11:07 -07:00
Ian Hickson
5fd259be24
Enable avoid_redundant_argument_values lint (#91409) 2021-10-07 20:13:02 -07:00
Anis Alibegić
a753d09cc9
Fixed several typos (#89485) 2021-09-07 14:56:04 -07:00
keyonghan
e311e3b3ef
skip staging update (#89137) 2021-08-30 09:26:05 -07:00
keyonghan
db4c104c8e
Use task name when uploading metrics to skia perf (#89004) 2021-08-26 22:31:02 -07:00
keyonghan
b67e264203
Revert "Skip staging test update to cocoon in test runner (#88835)" (#88971)
This reverts commit 884dfc260d5aea0c7bee16548df9ee031b8936e9.
2021-08-26 09:41:22 -07:00
Casey Hillers
b0950fd567
Revert "Use bucket to check staging test instead of builder name (#88908)" (#88969)
This reverts commit d36f246dcca6caa744e89171752cabba2a8d5807.
2021-08-26 09:19:50 -07:00
keyonghan
d36f246dcc
Use bucket to check staging test instead of builder name (#88908) 2021-08-25 20:36:02 -07:00
keyonghan
884dfc260d
Skip staging test update to cocoon in test runner (#88835) 2021-08-25 11:36:03 -07:00
keyonghan
b468be3030
Use default value for ResultData when uploading metrics from test runner (#88749) 2021-08-24 12:04:03 -07:00
keyonghan
208d312ff0
Upload devicelab test metrics from test runner (#88447) 2021-08-20 12:57:03 -07:00
yusufm
fe52e56614
Adding a timeout and retry to upload results step. (#87306)
* Adding a timeout and retry to upload results step.

* Adding tests for retries and timeout.

* Adding async to call

* Fixing sleep to trigger the timeout, and adding addition timeout test.

* Fixing minor typo.

* Fixing some minor flutter style issues.

Co-authored-by: yusufm <mohsinally@google.com>
2021-08-03 13:40:28 -07:00
Casey Hillers
79f377dabe
[devicelab] Only upload results on master (#87125) 2021-07-27 12:49:05 -07:00
Darren Austin
8fcace1d6c
Migrate devicelab tests and test runners to null safety. (#85999)
* Migrate devicelab tests and test runners to null safety.
2021-07-23 13:37:57 -07:00
keyonghan
3aeb794298
Extend test runner command to update test flaky status (#86513) 2021-07-19 09:46:03 -07:00