Coordinating the CI efforts for Linux + userspace
XDC 2021 - Virtual
Agenda
-
Trace-based testing: 1 hour
- Making traces is difficult and slow
- Confidential trace execution in a CI infra
- Reviewing frame outputs
- Automated frame reviewing for pre-merge testing
-
Kernel testing in GitLab: 1 hour
Challenges with trace-based testing
Making traces is time consuming!
Challenges with trace-based testing
Making traces is time consuming
Pick the right tool for the job
- Apitrace: OpenGL(ES) / DirectX 8-11 / DirectX 12 (WIP)
- GFXReconstruct: Vulkan
- Renderdoc: Single frame, OpenGL(ES) / DirectX 11-12 / Vulkan
Challenges with trace-based testing
Making traces is time consuming
Creating the trace
- Tracing real games running in Steam is difficult, requires many attempts
- Tracing can be super slow: framerates in frames per second
Challenges with trace-based testing
Making traces is time consuming
Selecting the relevant frames
- Find frames show-casing as much content as possible
- Make sure that the most important objects are the most visible
- ...
Rise of the Tomb Raider: Checking that the menu rendering is fine
Rise of the Tomb Raider: Close-up of Lara, along with the mountain in the distance. Only frame with her face in the entire benchmark
Rise of the Tomb Raider: Checking the snow motion blur, the lens flare, the snow effect on the jacket, and the metallic effect on her climbing gear
Rise of the Tomb Raider: New scene, more of the climbing gear, different outfit, lens flare, god rays, mist, reflections, ... Frame closest to Lara.
Rise of the Tomb Raider: New scene showing a river, trees, grass, some water splashes from a nearby waterfall, and some rain effect
Challenges with trace-based testing
Making traces is time consuming
Trimming the trace
- Why spend time on frames we did not select?
- Removing empty frames is notoriously hard...
- OpenGL work ongoing by Collabora
- Vulkan/GFXReconstruct is supposed to be designed for trimming
- DirectX: Anyone tried?
Challenges with trace-based testing
Confidential trace execution in a CI infra
Challenges with trace-based testing
Confidential trace execution in a CI infra
-
Challenges with traces of non-free games
- They contain non-redistributable art (textures, videos, ...)
- They can be quite big (> 1GB)
-
Requirements for the CI infra
- Requires caching on the test machines to avoid lengthy downloads
- The cache needs to be encrypted
Challenges with trace-based testing
Reviewing frame outputs
Challenges with trace-based testing
Reviewing frame outputs
Do you have what it takes to
know if a frame looks good?
Looking good?
Oops, missing head!
Better :)
Looking good?
The expected rendering
Wasn't too bad?
Looking good?
Let's see in context!
Challenges with trace-based testing
Automated frame reviewing
Challenges with trace-based testing
Automated frame reviewing
Sources of variance in rendering
- Floating point precision makes the order of operations in the shader important
- Any compiler change may change the output
- Inaccuracies can affect both the vertices and the fragments
- Can't simply allow for a +/- 1 change in any component...
- Order of shader executions is not guaranteed in GPUs
- Running the trace multiple times can lead to different outputs
Banding / dithering
Geometry noise - Diff
Geometry noise - Original img
Challenges with trace-based testing
Automated frame reviewing
How do we teach a machine was is good or not?
- Image checksums: Compare the frame to known-good ones
- Works for post-merge, if humans review new frames
- Not suitable for pre-merge because of variance
- Variance modelling: Document where in the image variance is expected, and how much
- This needs research, current strategy for Valve's CI
- Let our AI overlords figure it out?
Agenda
-
Trace-based testing: 1 hour
-
Kernel testing in GitLab: 1 hour
- Discussion
Kernel testing
Infra
Opens
- Can everyone use GitLab for their kernel tree?
- Where do we store the .gitlab-ci.yml?
- What should we test?
- How do we integrate with mailing lists?