320 Commits

Author SHA1 Message Date
liyuqian
552e74770d
Mark non-flaky test as such (#33269) 2019-05-23 12:39:55 -07:00
liyuqian
eb30745faf
Print warning if flutter drive is run in debug (#30747)
## Description

Print actionable warnings if `flutter drive` (where most of our performance benchmarks come from) is run in debug mode and it tries to gather benchmarks using `traceAction`.

## Related Issues

https://github.com/flutter/flutter/issues/30625

## Tests

I added the following tests:
* drive_perf_debug_warning devicelab test
2019-04-09 18:55:42 -07:00
liyuqian
5f060ba1c2
Mark cubic_bezier_perf__timeline_summary nonflaky (#30767) 2019-04-09 10:01:19 -07:00
Jonah Williams
1bfa2f2311
Intercept errors thrown by synchronous Completers in image resolution. (#30139) 2019-03-29 10:05:18 -07:00
Dan Field
ecfdd7e1ea
Detect and cleanup leaky processes (#29196)
* Detect and cleanup leaky processes
* Add flaky tests for detecting leaked processes
2019-03-22 14:32:36 -07:00
Jonah Williams
469a859c72
Enable code generation features in tool (#29399) 2019-03-21 13:12:10 -07:00
Dan Field
2a644f301e
Run non-perf sensitive tests on Cirrus (#27971)
* Run non-perf sensisitive tests on Cirrus
2019-03-10 07:52:44 -07:00
Jonah Williams
d4a441a0d9
Add integration to all targets (#29020) 2019-03-08 12:25:35 -08:00
liyuqian
19a6a6f431
Reland "Shader warm up (#27660)" (#28537)
This reverts commit adc8e159a548121a45ff7a45b4e556aff998d4bd.

This should be safe to land once https://github.com/flutter/flutter/pull/28530 gets merged

Merge on yellow doc test because the doc test is actually green.
2019-02-27 09:33:08 -08:00
liyuqian
2e3d9ea253
Mark non-flaky test as such (#28373) 2019-02-25 10:19:49 -08:00
liyuqian
adc8e159a5
Revert "Shader warm up (#27660)" (#28376)
This reverts commit a44f174efc218ac1b4b94b9d77dafdb5dfea1432.

Reason: start_up tests become flaky.

See https://github.com/flutter/flutter/issues/28374

TBR: xster

Merge on red to fix the tree
2019-02-22 17:32:37 -08:00
liyuqian
a44f174efc
Shader warm up (#27660)
This patch adds a default shader warm up process which moves shader compilation from the animation time to the startup time. This also provides an extension for `runApp` so developers can customize the warm up process.

This should reduce our worst_frame_rasterizer_time_millis from ~100ms to ~20-30ms for both flutter_gallery and complex_layout benchmarks. Besides, this should also have a significant improvement on 90th and 99th percentile time (50%-100% speedup in some cases, but I haven't tested them thoroughly; I'll let our device lab collect the data afterwards).

The tradeoff the is the startup time (time to first frame). Our `flutter run --profile --trace-startup` seems to be a little noisy and I see about 100ms-200ms increase in that measurement for complex_layout and flutter_gallery. Note that this only happens on the first run after install or data wipe. Later the Skia persistent cache will remove the overhead.

This also adds a cubic_bezier benchmark to test the custom shader warm up process.

This should fix https://github.com/flutter/flutter/issues/813 (either by `defaultShaderWarmUp`, or a `customShaderWarmUp`).
2019-02-22 15:37:02 -08:00
Jonah Williams
67cf21577f
Add basic codegen app to be used for integration testing and benchmarks (#27257) 2019-02-14 23:17:16 -08:00
liyuqian
de321b9cc1
Add scroll performance test for flutter_gallery (#27513)
Previously, we had a scroll_perf drive test but we were not exercising
it on our device lab. This helps us tracking issues like:
https://github.com/flutter/flutter/issues/24782
2019-02-04 22:54:11 -08:00
Dan Field
ff170ab5f3
mark tests as not flaky (#26334) 2019-01-09 18:55:42 -08:00
Amir Hardon
ff9f7236e6
Mark flutter_gallery__back_button_memory as flaky. (#25857)
See https://github.com/flutter/flutter/issues/25855
2018-12-28 15:34:15 -08:00
liyuqian
8ad4cbe06e
Add cull opacity perf test to device lab (#25381)
For https://github.com/flutter/flutter/issues/24712

This test verifies that https://github.com/flutter/engine/pull/6923 will speedup the average rasterize time of this test from ~150ms to ~10ms

Please see non-auto-generated files in 37b21d9fb4
2018-12-17 22:01:07 -08:00
liyuqian
7ecef3aa27
Mark flaky tests as such (#25482)
See https://github.com/flutter/flutter/issues/25481
2018-12-17 15:56:26 -08:00
Jonah Williams
d1deb32fe9
try disabling flutter run test (#25416) 2018-12-15 10:54:47 -08:00
Jonah Williams
f30029bae1
Run flutter tests through mini test engine when run directly (flutter run -t test_file) (#24930) 2018-12-12 12:51:16 -08:00
Dan Field
2729385758
Make host app test actually test host app (#24131)
* Make host app test actually test host app

* mark updated test as flaky
2018-11-09 19:09:56 -08:00
Dan Field
f1f4bda8da
Mark some tests as not flaky (#23840)
These tests have been run for a few days now with no flakiness.
2018-11-01 20:08:45 -07:00
Yegor
223826247f
deflake the dartdocs test (#23582) 2018-10-26 13:12:14 -07:00
Yegor
4452bd0a6b
Revert "Revert "make LocalHistoryRoute a proper super-mixin (#23382)"… (#23482)
* Revert "Revert "make LocalHistoryRoute a proper super-mixin (#23382)" (#23430)"

This reverts commit 3bbb3082b89ada4b0c44396148f3c2de731d1c6d.

This relands the LocalHistoryRoute change. The analyzer bug has been fixed.

* mark dartdocs as flaky
2018-10-25 09:56:23 -07:00
Michael Klimushyn
ce434022b5
Fix "E2E test setting and using isolate names (#23388)" (#23438)
Updates gradle.

This reverts commit d07c09c9a1a8bd9a0860c579154974437da14024.
2018-10-24 11:44:45 -07:00
Michael Goderbauer
55fe15bfdd
mark integration_ui_ios as flaky (#23437) 2018-10-23 16:45:17 -07:00
Michael Klimushyn
e70f19523b
Revert "E2E test setting and using isolate names (#23388)" (#23416)
This reverts commit 93573de2165c750fdeefcd2d620e2b8bd494fed6.

Fails in the devicelab:

```
run:stderr: A problem occurred evaluating project ':app'.
run:stderr: > Could not resolve all files for configuration 'classpath'.2018-10-23T10:46:46.335864:
run:stderr:    > Could not find aapt2-proto.jar (com.android.tools.build:aapt2-proto:0.3.1).2018-10-23T10:46:46.335960:
run:stderr:      Searched in the following locations:2018-10-23T10:46:46.336048:
```
2018-10-23 11:28:12 -07:00
Michael Klimushyn
93573de216
E2E test setting and using isolate names (#23388)
Adds an integration devicelab test that runs an Android app with two
custom named isolates. Tests that the isolate names are present and that
it's possible to attach to just one of the isolates.

Fixes flutter/flutter#22009
2018-10-23 09:30:00 -07:00
Michael Klimushyn
43c99404c5
Test flutter create --offline (#23278)
Verifies that flutter tools isn't rebuilt. Doesn't truly check that the
created output is runnable.
2018-10-22 16:39:28 -07:00
Greg Spencer
0ff9e8a928
Rename 'application' back to 'module', and make 'app' the default again for templates. (#22888)
We decided that redefining the default for templates was premature. We're going to go back to having "module" in experimental land again, and we'll try again when we have the feature set fully baked.

This keeps the writing of the .metadata files, and writing the template type to them, because that was a good improvement, and there are still a bunch of added tests that improve our coverage.
2018-10-10 11:01:40 -07:00
Greg Spencer
abb1758edf
Fix places in devicelab tests where it expects the app template. (#22696)
I missed some create template conversions in the devicelab directory.
2018-10-04 15:30:08 -07:00
Danny Tuppeny
55d70506ed
Mark module_test_ios as not-flaky (#22626)
The only reds on the dashboard for this test are on rows that have many failures across the run; this does not appear to be at all flaky.
2018-10-03 21:03:25 +01:00
Mikkel Nygaard Ravn
a600fe7f13 Support materializing Flutter module host app on iOS (#21276)
* Prototype

* Fix paths to Flutter library resources

* Invoke pod install as necessary for materialized modules

* Add devicelab test for module use on iOS

* Remove debug output

* Rebase, reame materialize editable

* Add devicelab test editable iOS host app

* Removed add2app test section
2018-09-25 15:21:13 -04:00
Ian Hickson
9f359aeee2
Remove all "flaky" annotations from devicelab (#21895)
These tests don't seem particularly more flaky than ony others any more.
2018-09-17 13:02:34 -07:00
Jonah Williams
c6e8a513f0
Add android semantics integration test to device lab (#20971) 2018-09-06 12:31:18 -07:00
amirh
146b906bbb
Mark the embedded_android_views_test as non flaky. (#20962)
Was marked as flaky as that's our policy for assing new device lab
tests. It's been consistently green so marking as non flaky.
2018-08-23 19:01:28 -07:00
amirh
1f31c3b354
Run the embedded Android views integration test on the device lab. (#20841) 2018-08-23 10:26:00 -07:00
Jonah Williams
f0e80933b0
Mark memory_nav as flaky (#20930) 2018-08-22 17:34:29 -07:00
Jonah Williams
452f255e95
undo changes to devicelab capabilities (#20536) 2018-08-13 13:26:32 -07:00
Jonah Williams
0892ace91e
agent capability doesnt work as expected (#20529) 2018-08-13 10:19:47 -07:00
Jonah Williams
cf88359ae4
restrict analysis task to faster machine (#20446) 2018-08-11 08:22:38 -07:00
liyuqian
e770685a4b
Mark complex_layout_scroll_perf__timeline_summary as non-flaky (#20272) 2018-08-08 10:25:36 -07:00
Greg Spencer
e60087a1a7
Remove Travis configuration from flutter/flutter (#20288)
This removes the final traces of Travis and Appveyor from the Flutter tree.

I've updated the documentation and fixed a couple of places where scripts look for Travis, and eliminated the dart tools runningOnTravis function (which was unused anyhow).

There are places in the flutter script that used to look for the environment variable TRAVIS. We actually do want to continue to detect that we're running on Travis there, since in the plugins repo we still use Travis (for the moment). In any case, it's OK, because the CI environment variable is set on all of the CI bots (Cirrus, Travis, and Appveyor).

FastLane doesn't have a setup_cirrus equivalent to setup_travis, but it actually doesn't matter there either, since it doesn't do Travis-specific things, and it also looks for the CI environment variable.
2018-08-07 13:41:33 -07:00
liyuqian
b6cd38d8df
Add tiles_scroll_perf__timeline_summary to devicelab (#20105) 2018-08-02 16:47:58 -07:00
Ryan Macnak
091da9c7d3 Revert "mark flaky benchmarks as flaky (#19492)"
This reverts commit 7a9dd709c6235153dd7500da65fb30dfbe76e560.

Bug: https://github.com/flutter/flutter/issues/19490
2018-08-01 14:15:22 -07:00
Chris Bracken
c0294e3a02
Eliminate Dart 1 benchmarks (#19430)
Flutter app development has been Dart 2 only for a significant amount of
time, and we're about to remove all support for Dart 1.
2018-07-18 15:56:51 -07:00
Michael Goderbauer
7a9dd709c6 mark flaky benchmarks as flaky (#19492) 2018-07-18 11:11:39 -07:00
Mikkel Nygaard Ravn
ca0e144f9f
Mark non-flaky tests as such (#18927) 2018-06-29 10:55:31 +02:00
Danny Tuppeny
e87b83bfd8 Mark flutter_gallery_instrumentation_test as flaky (#18887)
See #18879.
2018-06-27 15:48:22 -07:00
Ian Hickson
3ecd4301bf
Test performance of "flutter test" (#18810) 2018-06-26 17:34:25 -07:00