Putting Randomized Compiler Testing into Production
Sun 15 Nov 2020 15:20 - 15:40 at SPLASH-I - S-5 Chair(s): Davide Ancona, Eli Tilevich
We describe our experience over the last 15 months on a compiler testing technology transfer project: taking the GraphicsFuzz research project on randomized metamorphic testing of graphics shader compilers, and building the necessary tooling around it to provide a highly automated process for improving the Khronos Vulkan Conformance Test Suite (CTS) with test cases that expose fuzzer-found compiler bugs, or that plug gaps in test coverage. We present this tooling for test automation—gfauto—in detail, as well as our use of differential coverage and test case reduction as a method for automatically synthesising tests that fill coverage gaps. We explain the value that GraphicsFuzz has provided in automatically testing the ecosystem of tools for transforming, optimizing and validating Vulkan shaders, and the challenges brought by testing a tool ecosystem rather than a single tool. We discuss practical issues associated with putting automated metamorphic testing into production, and provide illustrative examples of bugs found during our work.
Sun 15 Nov Times are displayed in time zone: Central Time (US & Canada) change
15:00 - 16:20: S-5Research Papers at SPLASH-I +12h Chair(s): Davide AnconaDIBRIS, University of Genova, Italy, Eli TilevichVirginia Tech | |||
15:00 - 15:20 Talk | Model-View-Update-Communicate: Session Types meet the Elm Architecture Research Papers Simon FowlerUniversity of Glasgow Link to publication DOI Pre-print Media Attached | ||
15:20 - 15:40 Talk | Putting Randomized Compiler Testing into Production Research Papers Link to publication DOI Media Attached | ||
15:40 - 16:00 Talk | Owicki-Gries Reasoning for C11 RAR Research Papers Sadegh DalvandiUniversity of Surrey, Simon DohertyUniversity of Sheffield, Brijesh DongolUniversity of Surrey, Heike WehrheimPaderborn University Link to publication DOI Media Attached | ||
16:00 - 16:20 Talk | Test-Case Reduction via Test-Case Generation: Insights From the Hypothesis Reducer Research Papers Link to publication DOI Media Attached |
Mon 16 Nov Times are displayed in time zone: Central Time (US & Canada) change
03:00 - 04:20: S-5Research Papers at SPLASH-I Chair(s): Davide AnconaDIBRIS, University of Genova, Italy, Jeremy GibbonsDepartment of Computer Science, University of Oxford | |||
03:00 - 03:20 Talk | Model-View-Update-Communicate: Session Types meet the Elm Architecture Research Papers Simon FowlerUniversity of Glasgow Link to publication DOI Pre-print Media Attached | ||
03:20 - 03:40 Talk | Putting Randomized Compiler Testing into Production Research Papers Link to publication DOI Media Attached | ||
03:40 - 04:00 Talk | Owicki-Gries Reasoning for C11 RAR Research Papers Sadegh DalvandiUniversity of Surrey, Simon DohertyUniversity of Sheffield, Brijesh DongolUniversity of Surrey, Heike WehrheimPaderborn University Link to publication DOI Media Attached | ||
04:00 - 04:20 Talk | Test-Case Reduction via Test-Case Generation: Insights From the Hypothesis Reducer Research Papers Link to publication DOI Media Attached |