Testing

Cette page n'est pas encore traduite : la voici en anglais.

Aurore has a test harness: an app's screens run in a process with no window and no display, inputs are given through calls, time is simulated (frames advance without waiting, the same way every run, animations included), and screenshots are compared with references. It replaces scripts that drive a real window under Xvfb with xdotool and grab it with import.

It is built with -DAURORE_BUILD_TESTS=ON, on the desktop with the OpenGL 3 driver (the default). Without it, the library only has the few hooks the harness uses: a clock that can be simulated (setSimulatedTimeUsec, getWallTime), VideoContext::readPixels and Platform::setFactory.

The harness

library/include/aurore/test/harness.hpp, target Aurore::test.

#include <aurore.hpp>
#include <aurore/test/harness.hpp>

AURORE_TEST(settings_opens)
{
    my_app::install_hooks();          // what the app does before Application::init()
    aurore::test::Harness h;            // Application::init() and the window
    my_app::register_views();         // what it does after createWindow()
    h.push(new MainActivity());
    h.settle();                       // animations and tasks done

    AURORE_CHECK(h.focusByText("Settings"));   // presses Down until there
    h.tap(aurore::BUTTON_A);
    AURORE_CHECK_EQ(h.focusText(), std::string("Language"));
    AURORE_CHECK(h.cutTexts().empty());         // no label cut short ("…")
    AURORE_CHECK(h.matchesReference("settings"));
}

int main(int argc, char** argv) { return aurore::test::runTests(argc, argv); }

What it gives:

compareImages(actual, expected, tolerance) compares two images on its own: a pixel differs when a channel is off by more than tolerance, and the result has the count, the ratio, the largest difference and a map of the differences.

AURORE_TEST(name) declares a case, AURORE_CHECK(condition) and AURORE_CHECK_EQ(actual, expected) check (a failure is printed and the case goes on), aurore::test::runTests runs them: program runs every case, each in a process of its own, program name one, program --list lists them.

Screenshots and references

matchesReference("main") takes a screenshot, writes it to <output>/main.png, and compares it with <references>/main.png: it matches when at most maxRatio of the pixels (0.1 % by default) have a channel off by more than tolerance (8 by default). When it does not, <output>/main.diff.png shows the pixels that differ in red. A missing reference is written and matches, except in CI (the CI variable set, as on GitHub Actions) where it fails: a test whose reference was not committed, or whose name has a typo, does not pass unseen. AURORE_TEST_UPDATE=1 writes them all again, to commit after looking at them. Each case registered by aurore_add_tests stops after 5 minutes.

The folders: Options::referenceDir and outputDir, else AURORE_TEST_REFERENCES and AURORE_TEST_OUTPUT, else references and test-output in the working directory. aurore_add_tests (below) sets them.

The same frames are drawn on every run: the clock is simulated, the platform says the same thing everywhere (no battery or Wi-Fi shown, 127.0.0.1, the locale and theme of Options), the fonts are the app's resources: no system font is read unless Options::systemFonts asks for it (fonts). On Linux, the drawing is Mesa's software renderer (llvmpipe), with no display: the references are made with it, and a tolerance absorbs what another version of it changes in the antialiasing.

How it draws without a display

HeadlessPlatform (aurore/test/headless_platform.hpp) takes the place of GLFW or SDL (Platform::setFactory). Its video context draws into an offscreen framebuffer of an OpenGL 3.2 context made by EGL on Mesa's surfaceless platform: no X server, no Xvfb. Where EGL is missing (macOS, Windows), or with AURORE_HEADLESS_GLFW=1, the context is a hidden GLFW window's (which needs a display). On Ubuntu: libegl-dev to build, libegl1 and libgl1-mesa-dri to run.

The clock is borealis' own: getCPUTimeUsec() returns the simulated time once setSimulatedTimeUsec() set one, which the harness moves one frame's time before each frame. Animations, timers, aurore::delay, button repeats and the highlight read it; the main loop no longer sleeps to wait for a frame.

In an app

set(AURORE_BUILD_TESTS ON CACHE BOOL "" FORCE)   # or -DAURORE_BUILD_TESTS=ON
add_subdirectory(library/aurore/library)

enable_testing()
add_executable(my_app_tests tests/ui_tests.cpp ${MY_APP_SOURCES_BUT_MAIN})
target_link_libraries(my_app_tests PRIVATE Aurore::test)
aurore_add_tests(my_app_tests REFERENCES ${CMAKE_CURRENT_SOURCE_DIR}/tests/references)

aurore_add_tests registers each case with ctest (read from --list when ctest runs), with the references folder and <build>/test-output for what the tests got. The app's main() stays out of the test executable: what it does before and after createWindow() goes in functions the tests call too (the demo's demo/src/demo_app.cpp).

An Aurore installed with -DAURORE_INSTALL=ON -DAURORE_BUILD_TESTS=ON installs the harness too: after find_package(Aurore), Aurore::test and aurore_add_tests are there.

A smoke scenario of an Xvfb script (start, press keys, wait, grab the screen, compare) becomes a case: h.tap(aurore::BUTTON_DOWN, 3) for the key presses, h.settle() for the waits, matchesReference for the grab and the comparison, focusText() and cutTexts() for what was read in the log and in cut_text.txt.

The demo's tests

cmake -B build -G Ninja -DAURORE_PLATFORM=DESKTOP -DAURORE_WINDOW=GLFW -DAURORE_LIBROMFS=ON -DAURORE_BUILD_TESTS=ON
cmake --build build
ctest --test-dir build --output-on-failure

They need AURORE_LIBROMFS=ON, which configuring says: their reference screenshots are drawn with the resources built in, and from files the desktop build picks other fonts. With CMake 3.25 or later, cmake --workflow --preset tests configures, builds and runs them in build/tests.

tests/demo_tests.cpp: focus moves through the rail and into a tab, a dialog opens and closes, keyboard and touch, the window resized, redraw on demand and its debug mode, screenshots against tests/references/. CI runs them (linux (tests)).

tests/async_tests.cpp covers what crosses threads: a task coming back to a deleted view (View::aliveToken(), the ASYNC_* macros), the battery and Wi-Fi widgets read on the task thread (HeadlessPlatform::showStatus shows them), delay()/cancelDelay(), Threading::stop()/start(), events fired from another thread, logging from the task thread. Built with ThreadSanitizer they check the locking too:

cmake -B build-tsan -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DAURORE_PLATFORM=DESKTOP -DAURORE_WINDOW=GLFW -DAURORE_LIBROMFS=ON -DAURORE_BUILD_TESTS=ON \
    -DCMAKE_CXX_FLAGS=-fsanitize=thread -DCMAKE_C_FLAGS=-fsanitize=thread -DCMAKE_EXE_LINKER_FLAGS=-fsanitize=thread
cmake --build build-tsan --target aurore_async_tests
TSAN_OPTIONS=suppressions=$PWD/tests/tsan.supp ctest --test-dir build-tsan -R aurore_async_tests --output-on-failure

tests/tsan.supp leaves out what Mesa's software renderer, which draws the tests and is not built with the sanitizer, does in its own threads.

tests/host/: Cadran's plain C++ parts without a window, each suite linked only to the sources it tests: the palettes' contrast (every pair of cadran::palette::CONTRAST, both themes), safe text and the Markdown parser, ui::guarded, and the text helpers (cadran_text, format_text) over the real cadran.json catalogs. ctest -R host runs them alone.

The bench

aurore_bench (tests/bench.cpp) runs the demo's main screen under the harness with the Profiler on, over three scenarios (still, every frame drawn; moving through the rail; still with redraw on demand), and prints per frame: frames drawn, layout passes, views laid out, nanovg draw calls, allocations (a replaced operator new), and microseconds per phase (input, update, layout, draw, flush, swap, total).

aurore_bench                                   # the table
aurore_bench --json out.json                   # and the numbers
aurore_bench --write-baseline tests/bench_baseline.json
aurore_bench --check tests/bench_baseline.json # exit 1 on a regression

--check fails when a count is above the baseline's by more than --margin (25 %, and 1). Times depend on the machine: they are checked only with --time-margin <factor>. ctest runs the check.

The Profiler (aurore/core/profiler.hpp) is in every build and costs a test per phase while off. Profiler::setEnabled(true) turns it on (getLastFrame(), getFrameEvent(), setAllocationCounter()); AURORE_PROFILE=1 in the environment of a desktop app does too (a build with AURORE_DEV_TOOLS, see hot reload), and logs averages every 120 frames. The inspector's panel shows its figures, and it measures while the panel is open. Each FrameProfile has the time of each phase (events, input, update, layout, draw, inspector, flush, swap), the total and the interval since the iteration before, and counts: layouts, views laid out, nanovg's draw calls and triangles, allocations.

aurore-preview

aurore-preview resources/xml/tabs/layout.xml --theme light --lang fr --size 1920x1080 -o layout.png

Draws an XML layout to a PNG with the harness: --theme dark, light or a theme file (Theme::loadFile), --lang a locale, --size the window. The XML may use borealis', Cadran's and the demo's views; an app with views of its own builds the same tool on the harness after registering them.

Redraw on demand: finding a missing invalidation

With redraw on demand (Application::setRedrawOnDemand(true)), a view that changes how it looks without invalidate() or requestRedraw() stays stale on screen. Application::setRedrawDebug(true), or AURORE_REDRAW_DEBUG=1 on the desktop (a build with AURORE_DEV_TOOLS), draws each skipped frame all the same, without presenting it, compares it with the last frame presented, and logs a difference as a warning with its area and the views drawn there:

redraw debug: a skipped frame differs from the last one presented (412 pixels in 1104,20 96x28): no redraw asked for Label (id: clock)

Application::getMissedRedrawCount() and getLastMissedRedrawArea() give the same to a test. It reads every frame back: slow, for debugging, OpenGL drivers only (VideoContext::readPixels).

Modifier cette page sur GitHub