Fixes for Windows test failures on appveyor #5922
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
There was a problem hiding this comment.
Probably interesting, but I think this file is not actually related to this PR?
Member
There was a problem hiding this comment.
That's handy. There's also tools/test_triage.py that does something similar in PyQt4 in case you're not aware of it.
Contributor
Author
There was a problem hiding this comment.
yep, at least in should not belong to that commit... Also needed a fix for the image comparison decorator test images ...
Contributor
Author
There was a problem hiding this comment.
I left that in for the moment, should I add it or remove it? if "keep": in a different PR? I would run this script on failure on appveyor so that the html file is included in the result_images.zip, which is IMO nice for a quick overview...
Contributor
Author
|
The image comparison tests for some the failures are IMO to sensitive, at least I can't see much of a difference in that images. The only "real" difference is the font_styles, which is simply empty (-> all text missing)? |
Also fix the docstring to mention that it is part of miktex...
Also fixed a problem in an error message when bytes were appended to
a string (fails on py3.x) by using %.
One of the errors before:
======================================================================
ERROR: matplotlib.tests.test_backend_ps.test_savefig_to_stringio_with_usetex
----------------------------------------------------------------------
Traceback (most recent call last):
File "lib\site-packages\nose\case.py", line 198, in runTest
self.test(*self.arg)
File "lib\matplotlib\testing\decorators.py", line 152, in wrapped_callable
func(*args, **kwargs)
File "lib\matplotlib\testing\decorators.py", line 55, in failer
result = f(*args, **kwargs)
File "lib\matplotlib\tests\test_backend_ps.py", line 77, in test_savefig_to_stringio_with_usetex
_test_savefig_to_stringio()
File "lib\matplotlib\tests\test_backend_ps.py", line 40, in _test_savefig_to_stringio
fig.savefig(buffer, format=format)
File "lib\matplotlib\figure.py", line 1698, in savefig
self.canvas.print_figure(*args, **kwargs)
File "lib\matplotlib\backend_bases.py", line 2232, in print_figure
**kwargs)
File "lib\matplotlib\backends\backend_ps.py", line 985, in print_ps
return self._print_ps(outfile, 'ps', *args, **kwargs)
File "lib\matplotlib\backends\backend_ps.py", line 1012, in _print_ps
**kwargs)
File "lib\matplotlib\backends\backend_ps.py", line 1376, in _print_figure_tex
rotated=psfrag_rotated)
File "lib\matplotlib\backends\backend_ps.py", line 1539, in gs_distill
raise RuntimeError(m % output)
RuntimeError: ghostscript was not able to process your image.
Here is the full report generated by ghostscript:
b''
This is a script which I developed for ggplot https://github.com/yhat/ggplot/blob/master/visual_tests.py The script will go over the images in result_images and add them to a webpage, sorted by test files. At the top, there will be the test failures, so you can inspect the visually and try to figure out what's wrong with matplotlib (or the tests...) :-)
Contributor
Author
|
Current output in my local py35 test run: |
Contributor
Author
|
Ok, here are the tests from appveyor (which fail the build now): TARGET_ARCH=x86, CONDA_PY=27 TARGET_ARCH=x64, CONDA_PY=27 TARGET_ARCH=x64, CONDA_PY=34 TARGET_ARCH=x64, CONDA_PY=35 |
Member
|
The |
Contributor
Author
|
BTW, this seems to be out of sync: |
Member
Indeed it is. I'll open another PR for that. |
There seems to be an error here, so don't silence 7z too much and only filter out the individual files beeing compressed, but not the status messages.
* [ ]: matplotlib.tests.test_axes.test_specgram_freqs.test (RMS 0.042) (x64,35) * [ ]: matplotlib.tests.test_axes.test_specgram_freqs.test (RMS 0.042) (x64,35) * [ ]: matplotlib.tests.test_axes.test_specgram_magnitude_freqs.test (RMS 0.042) (x64,35) * [ ]: matplotlib.tests.test_axes.test_specgram_magnitude_freqs.test (RMS 0.042) (x64,35) -> before set to 0.03, set tolerance to 0.05 on windows
* [ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.006) (x64,35) * [ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.008) (x86,27) * [ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.012) (x64,27) * [ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.012) (x64,34) This has a black diff, so up the tolerance on windows to 0.013
Contributor
Author
|
There is also #5647, which lists these two tests as well:
-> currently set to 0.03, set tolerance to 0.05 on windows -> done
This has a black diff, so up the tolerance on windows to 0.013 -> done
This looks actually interesting: it seems that only the middle figure in the last row is different (at least the 32bit, py2.7 which I looked at) -> also done...
Also looks black, but only on py27/x86...? -> done
-> done |
affected: matplotlib.tests.test_patches.test_wedge_range.test (RMS 0.059) (x64,27) matplotlib.tests.test_patches.test_wedge_range.test (RMS 0.059) (x64,34) matplotlib.tests.test_patches.test_wedge_range.test (RMS 0.059) (x86,27) it seems that only the middle figure in the last row is different. Up the tolerance on windows to let the tests pass.
affected: * matplotlib.tests.test_axes.test_specgram_angle_freqs.test (RMS 0.002) (x86,27)
affected: * matplotlib.tests.test_triangulation.test_tri_smooth_gradient.test (RMS 0.014) (x64,35) The diff looks pitch black to me... -> up the tolerance...
Contributor
Author
|
https://travis-ci.org/matplotlib/matplotlib/jobs/105255492 [I really hate that the image comparison tests do not show the line number but only the decorator code path :-(] [Edit: seems to be a fluke, at least the next travis build didn't see that error?] |
Contributor
Author
|
And locally I still have the problem with |
Contributor
Author
|
I will open a new issue for my local test failure, that looks like the fallback font doesn't work on my system. [Edit: the problem was a stale font cache which prevent this test to find the fallback fonts... See #5929 for the story... Note to self: before writing issues, use a debugger...] |
Contributor
Author
|
Ok, the builds in my appveyor instance already build the two py2.7 versions of the tests and they are green: https://ci.appveyor.com/project/JanSchulz/matplotlib/build/1.0.135 |
Contributor
Author
|
Open question: should I remove the |
Contributor
Author
|
And it's green again (in my build) :-) |
tacaswell
added a commit
that referenced
this pull request
Feb 1, 2016
Fixes for Windows test failures on appveyor
Contributor
Author
|
@tacaswell @mdboom Should the tests for which I changed the tolerance be added to #5647? |
20 tasks
Member
|
@JanSchulz Yes it would be good to collect the information about non zero tolerance in #5647 |
Contributor
Author
|
@jenshnielsen ok, done directly in the first comment as additional Todos: #5647 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes the windows test errors as seen on appveyor and makes appveyor test failures into build failures.
Closes: #5748