Carl Worth [Thu, 11 Apr 2013 19:07:10 +0000 (12:07 -0700)]
Add --exact, --no-deps, or --no-prune to trim tests as needed
By explicitly specifying --exact, --no-deps, or --no-prune these tests
are now immune from any change in the default behavior of "apitrace
trim". Whether --exact or --auto is the default, these tests will get
exactly the behavior they want.
Carl Worth [Sat, 16 Feb 2013 01:19:57 +0000 (17:19 -0800)]
Rework testing of trimming of operations with no side effects
The previous test was bogus. The only call it trimmed was glxSwapBuffers
which is actually invalid, (this call definitely does have side effects,
but can easily be trimmed if desired with --trim-spec=drawing).
The new testing here uses a trace from a new program, (source
included), called few-side-effects which simply makes various calls
that do not have side effects, (such as glGetIntegerv and
glIsEnabled).
Then, the testing exercises the two independent mechanisms available
for trimming calls without side effects:
trim --prune
Trims calls within the user-specified range without side
effects.
trim --deps --trim-spec=no-side-effects
Trims calls outside the user-specific range without side
effects.
Three new tests are added here, one for each of the above cases and
one for both combined, (prune, trim, and prune-and-trim).
Carl Worth [Mon, 11 Mar 2013 22:45:05 +0000 (15:45 -0700)]
cli: Add test for "apitrace trim" with both --calls and --frames
This case was triggering a bug in "apitrace trim" where it would
incorrectly bail out after going past the last specified call (or
frame) without noticing that there were still further frames (or
calls) to be processed.
Carl Worth [Mon, 28 Jan 2013 10:55:12 +0000 (21:55 +1100)]
Fix test to correctly use cmake feature to find python executable
On my system, at least, there was no ${python} variable set, so the
tests for PIL were incorrectly failing, (such that many tests were not
being run at all).
Carl Worth [Tue, 29 Jan 2013 04:07:34 +0000 (15:07 +1100)]
cli: Add testing for recently-added "apitrace trim --trim-spec"
Each test here is quite simple. Without doing any image-based
verification, simply ensure that the expected calls are trimmed with
each different invocation of --trim-spec.
Carl Worth [Mon, 28 Jan 2013 12:05:26 +0000 (23:05 +1100)]
trim_stress: Only look at files matching [0-9]*.png when counting references
If there are any differences from a previous run, there will be
several additional image files left around, (.thumb.png, etc.). And we
don't want to count those.
Carl Worth [Mon, 28 Jan 2013 11:04:56 +0000 (22:04 +1100)]
trim_stress: Also test any traces in the directory trim_stress/local
This is useful for performing testing on traces that are too large to
be considered for include in git, or for traces that cannot be shared
due to proprietary contents.
Carl Worth [Mon, 28 Jan 2013 11:04:01 +0000 (22:04 +1100)]
trim_stress: Fail test at first frame which doesn't match reference
When stress-testing large traces, it's inefficient to work through the
entire trace before reporting an error that occurred in one of the
first frames.
Instead, check each frame one at a time immediately as each rendered
frame becomes available. Then fail the test at the first mismatched
frame.
Carl Worth [Mon, 28 Jan 2013 20:22:31 +0000 (07:22 +1100)]
Fix test to correctly use cmake feature to find python executable
On my system, at least, there was no ${python} variable set, so the
tests for PIL were incorrectly failing, (such that many tests were not
being run at all).
Carl Worth [Thu, 16 Aug 2012 20:39:10 +0000 (13:39 -0700)]
Track updates to "apitrace trim" which no longer drops glBindTexture calls
We still aren't sure *why* apitrace trim should have to leave these
calls around, but we have a trace which only passes the
trim-stress-test if these calls are untrimmed. While that mystery
waits to be solved, make the test here compatible with the current
behavior.
Carl Worth [Wed, 15 Aug 2012 18:16:52 +0000 (11:16 -0700)]
glxsimple: Introduce an inter-frame texture dependency to exercise trim code
The previous glxsimple program was too simple as each frame was
entirely self-contained, (textures were created, initialized, and
referenced only within single frames).
For more realistic trimming, we create and initialize a texture in one
frame, then reference that texture in a later frame. This is a better
test to ensure that when trimming the first frame the texture creation
and initialization is not discarded.
Carl Worth [Wed, 15 Aug 2012 18:05:13 +0000 (11:05 -0700)]
Add stress tests from "apitrace trim".
The trim_stress directory makes it simple to stress test "apitrace
trim" by simply dropping new trace files into this directory. The
stress testing ensures that creating a new trace by trimming to a
single frame still generates the same image as that frame from the
original trace. It performs this test exhaustively for each frame of
each trace.
This testing ensures that trimming doesn't break the rendered results
of any frame. It does not do anything to test that trimming actually
discards anything. So the existing cli tests are still
independendently useful for verifying that expected content is
discarded.
Carl Worth [Tue, 14 Aug 2012 23:50:00 +0000 (16:50 -0700)]
cli_driver: Rework driver script to preserve json-embedded whitespace
Previously, the line read from the script was being passed through
python's string.split() and string.join() functions. This was causing
sequences of successive whitespace characters to be collapsed down to
a single space, (in turn causing spurious failures when comparing
against expected output with successive whitespace).
Rework the driver to avoid munging the line past the separation of the
first command word.
Carl Worth [Mon, 13 Aug 2012 02:42:35 +0000 (19:42 -0700)]
trim-head: New test to ensure "apitrace trim" drops all calls after range
This test ensures that all calls beyond the user-specified range are
correctly dropped. It also ensures that uninteresting calls are
trimmed, (since the glxsimple trace includes many of those).
Carl Worth [Mon, 13 Aug 2012 00:50:04 +0000 (17:50 -0700)]
Make glxsimple loop over all drawing methods multiple times
We now have 3 different drawing methods, each performed two times, and
in a total of 6 different colors. Looping over theese things twice
means that we can trim out a single frame of any one method and ensure
that "apitrace trim" has some non-trivial trimming to do both before
and after the frame of interest.
Carl Worth [Mon, 13 Aug 2012 00:09:15 +0000 (17:09 -0700)]
Add the source of a very simple program using OpenGL through GLX.
We'll use this program to generate traces for use in the test suite,
(to exercise using "apitrace trim" to trim away unused textures,
shader programs, etc.).
There is a Makefile here, for convenience to any test writers wanting
to expand this program. Note that this Makefile us intentionally not
integrated with the cmake setup of the apitrace-tests project. That's
because it is not expected that this test program should be built or
run as a standard part of the test suite. Instead, useful trace files
generated from glxsimple will be directly committed to the
apitrace-tests repository.
Carl Worth [Mon, 13 Aug 2012 00:00:38 +0000 (17:00 -0700)]
cli: Use new --call-nos=no call to "apitrace dump-images" for consistent results
Recent "apitrace trim" is now dropping uninteresting calls by
default. This means that the call numbers of rendering operations are
changing. Consequently, image comparisons were failing not due to
image-content differences, but simply because the filenames (with
embedded call numbers in them) no longer matched.
We can avoid this problem by using the --call-nos=no option which was
added to "apitrace dump-images" in very recent versions.
Carl Worth [Thu, 9 Aug 2012 19:46:20 +0000 (12:46 -0700)]
Add a test to ensure "apitrace trim" performs proper dependency checking
Here we trim a trace by specifying (with --calls) only the final
glXSwapBuffers and then test that we can still produce a matching
image from the trace. This will only succeed if "apitrace trim"
performs dependency checking and includes many previous calls.
Of course, historically "apitrace trim" performed no such dependency
checks, so older versions of apitrace are expected to fail this test.
Carl Worth [Wed, 8 Aug 2012 23:18:21 +0000 (16:18 -0700)]
Add a test that "apitrace diff-images" flags mismatching images.
With this test we add a new EXPECT_FAILURE: attribute to the script
syntax to indicate that a particular apitrace command in the script is
expected to fail. This requires manually invoking the Popen
constructor rather than using subprocess.check_output, (which is
*close* to what we need but fails to assign the output when raising an
exception).
We also remove the globbing from the CMakeLists.txt file to get a
manual ordering of the tests (rather than executing in alphabetic
order).
Carl Worth [Wed, 8 Aug 2012 22:18:23 +0000 (15:18 -0700)]
Add a new 'cli' subdirectory for higher-level testing of apitrace CLI.
As described in the README, this new directory is for tests that
perform higher-level testing of the apitrace command-line interface
than either of the existing traces or apps directories.
Compared to traces, these new tests can do higher-level things like
dumping images from traces, comparing images, etc.
Compared to apps, these tests are intended to be portable across
multiple targets. (This isn't actually the case yet because commands
like "apitrace dump-images" currently work only on OpenGL. Also, even
when that is fixed, we will also need to then extend these tests to
have traces for each supported target for testing.)
In this commit, the only new test exercises "apitrace dump-images" and
"apitrace diff-images" on a simple trace file with one frame,
(tri.trace grabbed from apps/gl).
Carl Worth [Tue, 7 Aug 2012 22:47:53 +0000 (15:47 -0700)]
Add more README documentation on how existing test drivers are structured.
More than once, I've reverse-engineered how the test suite uses test
drivers (app_driver.py and tool_driver.py) to invoke scripts within
the apps/ and traces/ directory. In order to save my future-self some
work in doing this yet again, I've written some documentation in
README.markdown in each directory.
Hopefully this documentation will be useful for anyone wanting to
write additional tests.