Browse Source

Rewrite the IgnoreList logic to be more distutils-like

This should eliminate most of the inconsistencies with how distutils
treats MANIFEST.in files.  (Some inconsistencies remain: distutils
allows you to add back previously excluded files.)

The way I went about it:

- explicitly defined the internal canonical format for lists of file
  names

- changed it to use / instead of os.path.sep

- changed it to omit directories and list only files, which makes ignore
  handling simpler

To elaborate on the last point, here are some examples:

1. 'exclude foo' in MANIFEST.in works only on regular files,
   not directories, but how can we know if all we have is a string?

2. if you exclude all the files in a directory, distutils will omit that
   directory from the sdist, but it's difficult to do the same when all
   you have is a list of strings

Next, I changed IgnoreList to avoid fnmatch and be entirely based on
regular expressions, and this time I'm using distutils's own
translate_pattern() to produce exactly the same regular expressions
that distutils itself will use for matching.

This may introduce some surprising behaviour, like 'global-exclude a*.txt'
now matches on suffixes of file names, so it would exclude files named
bar.txt!  See https://bugs.python.org/issue14106.

Then tests started failing on Python 2.7, but all I get from the
__repr__ is a bunch of opaque <_sre.SRE_Pattern object at 0xdeadbeef>,
which are impossible to debug.  Therefore I'm declaring Python 2.7
support officially dropped!

check-manifests's own configuration (--ignore or [check-manifest]
ignore=) is now treated as a global-exclude statement in MANIFEST.in,
which means:

- it's matched anywhere in the file tree
- it's matched against a filename suffix due to the above-mentioned
  Python bug
- it's ignored if it matches a directory

You can ignore directories only by ignoring every file inside it.
There's no way currently of specifying that that works for arbitrarily
nested trees, but you can try --ignore=dir/*,dir/*/*,dir/*/*/*
until you get the result you want.

This decision is not cast in stone: I may in the future change the
handling of --ignore to match files and directories, because there's no
reason it has to be distutils-compatible.

Expect regressions on Windows.  I'll fix any that I find.
pull/117/head
Marius Gedminas 2 years ago
parent
commit
f56a51f2b2
  1. 1
      .gitignore
  2. 2
      .travis.yml
  3. 22
      CHANGES.rst
  4. 1
      appveyor.yml
  5. 238
      check_manifest.py
  6. 4
      setup.py
  7. 404
      tests.py
  8. 2
      tox.ini

1
.gitignore vendored

@ -10,3 +10,4 @@ tmp/
build/
tags
coverage.xml
.mypy_cache/

2
.travis.yml

@ -1,12 +1,10 @@
language: python
cache: pip
python:
- 2.7
- 3.5
- 3.6
- 3.7
- 3.8
- pypy
- pypy3
env:
- FORCE_TEST_VCS=bzr

22
CHANGES.rst

@ -8,6 +8,28 @@ Changelog
- Added ``-q``/``--quiet`` command line argument. This will reduce the verbosity
of informational output, e.g. for use in a CI pipeline.
- Rewrote the ignore logic to be more compatible with distutils. This might
have introduced some regressions, so please file bugs! One side effect of
this is that ``--ignore`` (or the ``ignore`` setting in the config file)
is now handled the same way as ``global-exclude`` in a ``MANIFEST.in``, which
means:
- it's matched anywhere in the file tree
- it's matched against a filename *suffix* rather than an entire filename,
due to https://bugs.python.org/issue14106
- it's ignored if it matches a directory
You can ignore directories only by ignoring every file inside it.
There's no way currently of specifying that that works for arbitrarily
nested trees, but you can try ``--ignore=dir/*,dir/*/*,dir/*/*/*``
until you get the result you want.
This decision is not cast in stone: I may in the future change the
handling of ``--ignore`` to match files and directories, because there's no
reason it has to be distutils-compatible.
- Drop Python 2.7 support.
0.41 (2020-02-25)
-----------------

1
appveyor.yml

@ -4,7 +4,6 @@ environment:
matrix:
# https://www.appveyor.com/docs/installed-software#python lists available
# versions
- PYTHON: "C:\\Python27"
- PYTHON: "C:\\Python35"
- PYTHON: "C:\\Python36"
- PYTHON: "C:\\Python37"

238
check_manifest.py

@ -30,8 +30,8 @@ import tarfile
import tempfile
import unicodedata
import zipfile
from distutils.filelist import glob_to_re
from contextlib import contextmanager, closing
from distutils.filelist import translate_pattern
from distutils.text_file import TextFile
try:
@ -266,16 +266,39 @@ def get_one_file_in(dirname):
#
# - contain Unicode filenames (normalized to NFC on OS X)
# - be sorted
# - use os.path.sep as the directory separator
# (I'm not sure this is done correctly at the moment!)
# - list directories as well as files (but without trailing slashes)
# - use / as the directory separator
# - list only files, but not directories
#
# We get these file lists from various sources (zip files, tar files, version
# control systems) and we have to normalize them into our common format before
# comparing
# comparing.
#
def canonical_file_list(filelist):
"""Return the file list convered to a canonical form.
This means:
- converted to Unicode normal form C, when running on Mac OS X
- sorted alphabetically
- use / as the directory separator
- list files but not directories
Caveat: since it works on file lists taken from archives and such, it
doesn't know whether a particular filename refers to a file or a directory,
unless it finds annother filename that is inside the first one. In other
words, canonical_file_list() will not remove the names of empty directories
if those appear in the initial file list.
"""
names = set(normalize_names(filelist))
for name in list(names):
while name:
name = posixpath.dirname(name)
names.discard(name)
return sorted(names)
def get_sdist_file_list(sdist_filename, ignore):
"""Return the list of interesting files in a source distribution.
@ -284,27 +307,9 @@ def get_sdist_file_list(sdist_filename, ignore):
Supports .tar.gz and .zip sdists.
"""
# The sorted() is probably redundant here, given that
# get_archive_file_list() returns a sorted list, but I'm not 100% sure
# normalize_names() won't affect the sort order.
# TODO: move normalize_names() into get_archive_file_list(), because
# get_vcs_files() first normalizes and then calls
# add_directories_and_sort().
return sorted(normalize_names(strip_sdist_extras(ignore,
strip_toplevel_name(get_archive_file_list(sdist_filename)))))
def unicodify(filename):
"""Make sure filename is Unicode.
Because the tarfile module on Python 2 doesn't return Unicode.
"""
if isinstance(filename, bytes):
# XXX: Ah, but is it right to use the locale encoding here, or should I
# use sys.getfilesystemencoding()? A good question!
return filename.decode(locale.getpreferredencoding())
else:
return filename
return strip_sdist_extras(
ignore,
strip_toplevel_name(get_archive_file_list(sdist_filename)))
def get_archive_file_list(archive_filename):
@ -321,10 +326,20 @@ def get_archive_file_list(archive_filename):
else:
raise Failure('Unrecognized archive type: %s'
% os.path.basename(archive_filename))
# Hm, should we normalize_names() here? Maybe, maybe not. The only caller
# of get_archive_file_list() is get_sdist_file_list(), and it calls the
# normalize_names().
return add_directories_and_sort(filelist)
return canonical_file_list(filelist)
def unicodify(filename):
"""Make sure filename is Unicode.
Because the tarfile module on Python 2 doesn't return Unicode.
"""
if isinstance(filename, bytes):
# XXX: Ah, but is it right to use the locale encoding here, or should I
# use sys.getfilesystemencoding()? A good question!
return filename.decode(locale.getpreferredencoding())
else:
return filename
def strip_toplevel_name(filelist):
@ -360,8 +375,7 @@ def strip_toplevel_name(filelist):
def add_prefix_to_each(prefix, filelist):
"""Add a prefix to each name in a file list.
>>> filelist = add_prefix_to_each('foo/bar', ['a', 'b', 'c/d'])
>>> [f.replace(os.path.sep, '/') for f in filelist]
>>> add_prefix_to_each('foo/bar', ['a', 'b', 'c/d'])
['foo/bar/a', 'foo/bar/b', 'foo/bar/c/d']
"""
@ -539,12 +553,12 @@ def detect_vcs(ui):
def get_vcs_files(ui):
"""List all files under version control in the current directory."""
vcs = detect_vcs(ui)
return add_directories_and_sort(normalize_names(vcs.get_versioned_files()))
return canonical_file_list(vcs.get_versioned_files())
def normalize_names(names):
"""Normalize file names."""
return list(map(normalize_name, names))
return [normalize_name(name) for name in names]
def normalize_name(name):
@ -555,8 +569,8 @@ def normalize_name(name):
And encodings may trip us up too, especially when comparing lists
of files. Plus maybe lowercase versus uppercase.
"""
name = os.path.normpath(name)
name = unicodify(name)
name = os.path.normpath(name).replace(os.path.sep, '/')
name = unicodify(name) # XXX is this necessary?
if sys.platform == 'darwin':
# Mac OS X may have problems comparing non-ASCII filenames, so
# we convert them.
@ -564,120 +578,77 @@ def normalize_name(name):
return name
def add_directories_and_sort(names):
"""Git/Mercurial/zip files omit directories, let's add them back."""
# Let's make sure we iterate names once, in case it's an iterator and not a
# list.
res = list(names)
seen = set(res)
for name in list(res):
while True:
name = os.path.dirname(name)
if not name or name in seen:
break
res.append(name)
seen.add(name)
return sorted(res)
#
# Packaging logic
#
class IgnoreList(object):
def __init__(self, ignore=(), _ignore_regexps=()):
self.ignore = list(ignore)
self.ignore_regexps = list(_ignore_regexps)
def __init__(self):
self._regexps = []
@classmethod
def default(cls):
return cls(DEFAULT_IGNORE)
return (
cls()
# these are always generated
.global_exclude('PKG-INFO')
.global_exclude('*.egg-info/*')
# setup.cfg is always generated, but sometimes also kept in source control
.global_exclude('setup.cfg')
# it's not a problem if the sdist is lacking these files:
.global_exclude(
'.hgtags', '.hgsigs', '.hgignore', '.gitignore', '.bzrignore',
'.gitattributes',
)
# GitHub template files
.prune('.github')
# we can do without these in sdists
.global_exclude('.travis.yml')
.global_exclude('Jenkinsfile')
# It's convenient to ship compiled .mo files in sdists, but they
# shouldn't be checked in, so don't complain that they're missing
# from VCS
.global_exclude('*.mo')
)
def clear(self):
self.ignore = []
self.ignore_regexps = []
self._regexps = []
def __repr__(self):
return 'IgnoreList(%r, %r)' % (self.ignore, self.ignore_regexps)
return 'IgnoreList(%r)' % (self._regexps)
def __eq__(self, other):
return (isinstance(other, IgnoreList) and self.ignore == other.ignore
and self.ignore_regexps == other.ignore_regexps)
return isinstance(other, IgnoreList) and self._regexps == other._regexps
def __iadd__(self, other):
assert isinstance(other, IgnoreList)
self.ignore += other.ignore
self.ignore_regexps += other.ignore_regexps
self._regexps += other._regexps
return self
def __add__(self, other):
if not isinstance(other, IgnoreList):
return NotImplemented
result = IgnoreList()
result.ignore = self.ignore + other.ignore
result.ignore_regexps = self.ignore_regexps + other.ignore_regexps
return result
def exclude(self, *patterns):
# An exclude of 'dirname/*css' can match 'dirname/foo.css'
# but not 'dirname/subdir/bar.css'. We need a regular
# expression for that, since fnmatch doesn't pay attention to
# directory separators.
for pat in patterns:
if '*' in pat or '?' in pat or '[!' in pat:
self.ignore_regexps.append(_glob_to_regexp(pat))
else:
self.ignore.append(pat)
self._regexps.append(translate_pattern(pat, anchor=True))
return self
def global_exclude(self, *patterns):
for pat in patterns:
self.ignore.append(pat)
self._regexps.append(translate_pattern(pat, anchor=False))
return self
def recursive_exclude(self, dirname, *patterns):
# Strip path separator for clarity.
dirname = dirname.rstrip('/\\')
for pattern in patterns:
if pattern.startswith('*'):
self.ignore.append(dirname + os.path.sep + pattern)
else:
# 'recursive-exclude plone metadata.xml' should
# exclude plone/metadata.xml and
# plone/*/metadata.xml, where * can be any number
# of sub directories. We could use a regexp, but
# two ignores seems easier.
self.ignore.append(dirname + os.path.sep + pattern)
self.ignore.append(
dirname + os.path.sep + '*' + os.path.sep + pattern)
for pat in patterns:
self._regexps.append(translate_pattern(pat, prefix=dirname))
return self
def prune(self, subdir):
subdir = subdir.rstrip('/\\')
self.ignore.append(subdir)
self.ignore.append(subdir + os.path.sep + '*')
self._regexps.append(translate_pattern(None, prefix=subdir))
return self
def filter(self, filelist):
return [name for name in filelist
if not file_matches(name, self.ignore)
and not file_matches_regexps(name, self.ignore_regexps)]
# it's fine if any of these are missing in the VCS or in the sdist
DEFAULT_IGNORE = [
'PKG-INFO', # always generated
'*.egg-info', # always generated
'*.egg-info/*', # always generated
'setup.cfg', # always generated, sometimes also kept in source control
# it's not a problem if the sdist is lacking these files:
'.hgtags', '.hgsigs', '.hgignore', '.gitignore', '.bzrignore',
'.gitattributes',
'.github', # GitHub template files
'.github/*', # GitHub template files
'.travis.yml',
'Jenkinsfile',
# it's convenient to ship compiled .mo files in sdists, but they shouldn't
# be checked in
'*.mo',
]
if not any(rx.search(name) for rx in self._regexps)]
WARN_ABOUT_FILES_IN_VCS = [
# generated files should not be committed into the VCS
@ -692,9 +663,7 @@ WARN_ABOUT_FILES_IN_VCS = [
'.#*',
]
_sep = re.escape(os.path.sep)
SUGGESTIONS = [(re.compile(pattern.replace('/', _sep)), suggestion) for pattern, suggestion in [
SUGGESTIONS = [(re.compile(pattern), suggestion) for pattern, suggestion in [
# regexp -> suggestion
('^([^/]+[.](cfg|ini))$', r'include \1'),
('^([.]travis[.]yml)$', r'include \1'),
@ -793,21 +762,10 @@ def read_manifest(ui):
return _get_ignore_from_manifest('MANIFEST.in', ui)
def _glob_to_regexp(pat):
"""Compile a glob pattern into a regexp.
We need to do this because fnmatch allows * to match /, which we
don't want. E.g. a MANIFEST.in exclude of 'dirname/*css' should
match 'dirname/foo.css' but not 'dirname/subdir/bar.css'.
"""
return glob_to_re(pat)
def _get_ignore_from_manifest(filename, ui):
"""Gather the various ignore patterns from a MANIFEST.in.
Returns a list of standard ignore patterns and a list of regular
expressions to ignore.
Returns an IgnoreList instance.
"""
class MyTextFile(TextFile):
@ -838,8 +796,7 @@ def _get_ignore_from_manifest_lines(lines, ui):
'lines' should be a list of strings with comments removed
and continuation lines joined.
Returns a list of standard ignore patterns and a list of regular
expressions to ignore.
Returns an IgnoreList instance.
"""
ignore = IgnoreList()
for line in lines:
@ -890,11 +847,6 @@ def file_matches(filename, patterns):
for pat in patterns)
def file_matches_regexps(filename, patterns):
"""Does this filename match any of the regular expressions?"""
return any(re.match(pat, filename) for pat in patterns)
def strip_sdist_extras(ignore, filelist):
"""Strip generated files that are only present in source distributions.
@ -911,7 +863,11 @@ def find_bad_ideas(filelist):
def find_suggestions(filelist):
"""Suggest MANIFEST.in patterns for missing files."""
"""Suggest MANIFEST.in patterns for missing files.
Returns two lists: one with suggested MANIGEST.in commands, and one with
files for which no suggestions were offered.
"""
suggestions = set()
unknowns = []
for filename in filelist:

4
setup.py

@ -42,8 +42,6 @@ setup(
'License :: OSI Approved :: MIT License',
'Operating System :: OS Independent',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
@ -56,7 +54,7 @@ setup(
py_modules=['check_manifest'],
zip_safe=False,
python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
python_requires=">=3.5",
install_requires=['toml', 'pep517'],
extras_require={
'test': ['mock'],

404
tests.py

@ -4,6 +4,7 @@ import codecs
import doctest
import locale
import os
import posixpath
import shutil
import subprocess
import sys
@ -219,7 +220,7 @@ class Tests(unittest.TestCase):
filename = os.path.join(self.make_temp_dir(), 'archive.zip')
self.create_zip_file(filename, ['a', 'b/c'])
self.assertEqual(get_archive_file_list(filename),
['a', 'b', 'b/c'])
['a', 'b/c'])
def test_get_archive_file_list_zip_nonascii(self):
from check_manifest import get_archive_file_list
@ -234,7 +235,7 @@ class Tests(unittest.TestCase):
filename = os.path.join(self.make_temp_dir(), 'archive.tar')
self.create_tar_file(filename, ['a', 'b/c'])
self.assertEqual(get_archive_file_list(filename),
['a', 'b', 'b/c'])
['a', 'b/c'])
def test_get_archive_file_list_tar_nonascii(self):
from check_manifest import get_archive_file_list
@ -287,32 +288,29 @@ class Tests(unittest.TestCase):
self.assertEqual(normalize_names(["a", j("b", ""), j("c", "d"),
j("e", "f", ""),
j("g", "h", "..", "i")]),
["a", "b", j("c", "d"), j("e", "f"), j("g", "i")])
["a", "b", "c/d", "e/f", "g/i"])
def test_add_directories_and_sort(self):
from check_manifest import add_directories_and_sort
def test_canonical_file_list(self):
from check_manifest import canonical_file_list
j = os.path.join
self.assertEqual(
add_directories_and_sort(['a', 'b', j('c', 'd'), j('e', 'f')]),
['a', 'b', 'c', j('c', 'd'), 'e', j('e', 'f')])
canonical_file_list(['b', 'a', 'c', j('c', 'd'), j('e', 'f'),
'g', j('g', 'h', 'i', 'j')]),
['a', 'b', 'c/d', 'e/f', 'g/h/i/j'])
def test_file_matches(self):
from check_manifest import file_matches
# On Windows we might get the pattern list from setup.cfg using / as
# the directory separator, but the filenames we're matching against
# will use os.path.sep
patterns = ['setup.cfg', '*.egg-info', '*.egg-info/*']
j = os.path.join
self.assertFalse(file_matches('setup.py', patterns))
self.assertTrue(file_matches('setup.cfg', patterns))
self.assertTrue(file_matches(j('src', 'zope.foo.egg-info'), patterns))
self.assertTrue(
file_matches(j('src', 'zope.foo.egg-info', 'SOURCES.txt'),
patterns))
self.assertTrue(file_matches('src/zope.foo.egg-info', patterns))
self.assertTrue(file_matches('src/zope.foo.egg-info/SOURCES.txt',
patterns))
def test_strip_sdist_extras(self):
from check_manifest import strip_sdist_extras, IgnoreList
filelist = list(map(os.path.normpath, [
from check_manifest import canonical_file_list
filelist = canonical_file_list([
'.github',
'.github/ISSUE_TEMPLATE',
'.github/ISSUE_TEMPLATE/bug_report.md',
@ -331,8 +329,8 @@ class Tests(unittest.TestCase):
'src/zope/foo/language.mo',
'src/zope.foo.egg-info',
'src/zope.foo.egg-info/SOURCES.txt',
]))
expected = list(map(os.path.normpath, [
])
expected = canonical_file_list([
'setup.py',
'README.txt',
'src',
@ -341,12 +339,13 @@ class Tests(unittest.TestCase):
'src/zope/foo',
'src/zope/foo/__init__.py',
'src/zope/foo/language.po',
]))
])
ignore = IgnoreList.default()
self.assertEqual(strip_sdist_extras(ignore, filelist), expected)
def test_strip_sdist_extras_with_manifest(self):
from check_manifest import strip_sdist_extras, IgnoreList
from check_manifest import canonical_file_list
from check_manifest import _get_ignore_from_manifest_lines
manifest_in = textwrap.dedent("""
graft src
@ -355,7 +354,7 @@ class Tests(unittest.TestCase):
prune src/dump
recursive-exclude src/zope *.sh
""")
filelist = list(map(os.path.normpath, [
filelist = canonical_file_list([
'.github/ISSUE_TEMPLATE/bug_report.md',
'.gitignore',
'setup.py',
@ -377,8 +376,8 @@ class Tests(unittest.TestCase):
'src/zope/foo/foohelper.sh',
'src/zope.foo.egg-info',
'src/zope.foo.egg-info/SOURCES.txt',
]))
expected = list(map(os.path.normpath, [
])
expected = canonical_file_list([
'setup.py',
'MANIFEST.in',
'README.txt',
@ -390,7 +389,7 @@ class Tests(unittest.TestCase):
'src/zope/foo/__init__.py',
'src/zope/foo/language.po',
'src/zope/foo/config.cfg',
]))
])
ignore = IgnoreList.default()
ignore += _get_ignore_from_manifest_lines(manifest_in.splitlines(), self.ui)
result = strip_sdist_extras(ignore, filelist)
@ -474,80 +473,39 @@ class Tests(unittest.TestCase):
self.assertEqual(e('dist/foo_bar-1.2.3.dev4+g12345.zip'), '1.2.3.dev4+g12345')
self.assertEqual(e('dist/foo_bar-1.2.3.dev4+g12345.tar.gz'), '1.2.3.dev4+g12345')
def test_glob_to_regexp(self):
from check_manifest import _glob_to_regexp as g2r
sep = os.path.sep.replace('\\', '\\\\')
if sys.version_info >= (3, 7):
self.assertEqual(g2r('foo.py'), r'(?s:foo\.py)\Z')
self.assertEqual(g2r('foo/bar.py'), r'(?s:foo/bar\.py)\Z')
self.assertEqual(g2r('foo*.py'), r'(?s:foo[^%s]*\.py)\Z' % sep)
self.assertEqual(g2r('foo?.py'), r'(?s:foo[^%s]\.py)\Z' % sep)
self.assertEqual(g2r('foo[123].py'), r'(?s:foo[123]\.py)\Z')
self.assertEqual(g2r('foo[!123].py'), r'(?s:foo[^123]\.py)\Z')
self.assertEqual(g2r('foo/*.py'), r'(?s:foo/[^%s]*\.py)\Z' % sep)
elif sys.version_info >= (3, 6):
self.assertEqual(g2r('foo.py'), r'(?s:foo\.py)\Z')
self.assertEqual(g2r('foo/bar.py'), r'(?s:foo\/bar\.py)\Z')
self.assertEqual(g2r('foo*.py'), r'(?s:foo[^%s]*\.py)\Z' % sep)
self.assertEqual(g2r('foo?.py'), r'(?s:foo[^%s]\.py)\Z' % sep)
self.assertEqual(g2r('foo[123].py'), r'(?s:foo[123]\.py)\Z')
self.assertEqual(g2r('foo[!123].py'), r'(?s:foo[^123]\.py)\Z')
self.assertEqual(g2r('foo/*.py'), r'(?s:foo\/[^%s]*\.py)\Z' % sep)
else:
self.assertEqual(g2r('foo.py'), r'foo\.py\Z(?ms)')
self.assertEqual(g2r('foo/bar.py'), r'foo\/bar\.py\Z(?ms)')
self.assertEqual(g2r('foo*.py'), r'foo[^%s]*\.py\Z(?ms)' % sep)
self.assertEqual(g2r('foo?.py'), r'foo[^%s]\.py\Z(?ms)' % sep)
self.assertEqual(g2r('foo[123].py'), r'foo[123]\.py\Z(?ms)')
self.assertEqual(g2r('foo[!123].py'), r'foo[^123]\.py\Z(?ms)')
self.assertEqual(g2r('foo/*.py'), r'foo\/[^%s]*\.py\Z(?ms)' % sep)
def test_get_ignore_from_manifest_lines(self):
from check_manifest import _get_ignore_from_manifest_lines
from check_manifest import _glob_to_regexp as g2r
from check_manifest import IgnoreList
j = os.path.join
parse = partial(_get_ignore_from_manifest_lines, ui=self.ui)
# The return value is a tuple with two lists:
# ([<list of filename ignores>], [<list of regular expressions>])
self.assertEqual(parse([]),
IgnoreList([], []))
IgnoreList())
self.assertEqual(parse(['', ' ']),
IgnoreList([], []))
IgnoreList())
self.assertEqual(parse(['exclude *.cfg']),
IgnoreList([], [g2r('*.cfg')]))
IgnoreList().exclude('*.cfg'))
self.assertEqual(parse(['exclude *.cfg']),
IgnoreList([], [g2r('*.cfg')]))
IgnoreList().exclude('*.cfg'))
self.assertEqual(parse(['\texclude\t*.cfg foo.* bar.txt']),
IgnoreList(['bar.txt'], [g2r('*.cfg'), g2r('foo.*')]))
IgnoreList().exclude('*.cfg', 'foo.*', 'bar.txt'))
self.assertEqual(parse(['exclude some/directory/*.cfg']),
IgnoreList([], [g2r('some/directory/*.cfg')]))
IgnoreList().exclude('some/directory/*.cfg'))
self.assertEqual(parse(['include *.cfg']),
IgnoreList([], []))
IgnoreList())
self.assertEqual(parse(['global-exclude *.pyc']),
IgnoreList(['*.pyc'], []))
IgnoreList().global_exclude('*.pyc'))
self.assertEqual(parse(['global-exclude *.pyc *.sh']),
IgnoreList(['*.pyc', '*.sh'], []))
IgnoreList().global_exclude('*.pyc', '*.sh'))
self.assertEqual(parse(['recursive-exclude dir *.pyc']),
IgnoreList([j('dir', '*.pyc')], []))
IgnoreList().recursive_exclude('dir', '*.pyc'))
self.assertEqual(parse(['recursive-exclude dir *.pyc foo*.sh']),
IgnoreList([j('dir', '*.pyc'), j('dir', 'foo*.sh'),
j('dir', '*', 'foo*.sh')], []))
IgnoreList().recursive_exclude('dir', '*.pyc', 'foo*.sh'))
self.assertEqual(parse(['recursive-exclude dir nopattern.xml']),
IgnoreList([j('dir', 'nopattern.xml'),
j('dir', '*', 'nopattern.xml')], []))
IgnoreList().recursive_exclude('dir', 'nopattern.xml'))
# We should not fail when a recursive-exclude line is wrong:
self.assertEqual(parse(['recursive-exclude dirwithoutpattern']),
IgnoreList([], []))
IgnoreList())
self.assertEqual(parse(['prune dir']),
IgnoreList(['dir', j('dir', '*')], []))
# You should not add a slash at the end of a prune, but let's
# not fail over it or end up with double slashes.
self.assertEqual(parse(['prune dir/']),
IgnoreList(['dir', j('dir', '*')], []))
# You should also not have a leading slash
self.assertEqual(parse(['prune /dir']),
IgnoreList(['/dir', j('/dir', '*')], []))
IgnoreList().prune('dir'))
# And a mongo test case of everything at the end
text = textwrap.dedent("""
exclude *.02
@ -563,24 +521,26 @@ class Tests(unittest.TestCase):
""").splitlines()
self.assertEqual(
parse(text),
IgnoreList([
'bar.txt',
'*.10',
'*.11',
'*.12',
'30',
j('30', '*'),
j('40', '*.41'),
j('42', '*.43'),
j('42', '44.*'),
j('42', '*', '44.*'),
], [
g2r('*.02'),
g2r('*.03'),
g2r('04.*'),
g2r('*.05'),
g2r('some/directory/*.cfg'),
]))
IgnoreList()
.exclude('*.02', '*.03', '04.*', 'bar.txt', '*.05', 'some/directory/*.cfg')
.global_exclude('*.10', '*.11', '*.12')
.prune('30')
.recursive_exclude('40', '*.41')
.recursive_exclude('42', '*.43', '44.*')
)
def test_get_ignore_from_manifest_lines_warns(self):
from check_manifest import _get_ignore_from_manifest_lines, IgnoreList
parse = partial(_get_ignore_from_manifest_lines, ui=self.ui)
text = textwrap.dedent("""
graft a/
recursive-include /b *.txt
""").splitlines()
self.assertEqual(parse(text), IgnoreList())
self.assertEqual(self.ui.warnings, [
'ERROR: Trailing slashes are not allowed in MANIFEST.in on Windows: a/',
'ERROR: Leading slashes are not allowed in MANIFEST.in on Windows: /b',
])
def test_get_ignore_from_manifest(self):
from check_manifest import _get_ignore_from_manifest, IgnoreList
@ -595,7 +555,7 @@ class Tests(unittest.TestCase):
'''))
ui = MockUI()
self.assertEqual(_get_ignore_from_manifest(filename, ui),
IgnoreList(['test.dat']))
IgnoreList().exclude('test.dat'))
self.assertEqual(ui.warnings, [])
def test_get_ignore_from_manifest_warnings(self):
@ -607,7 +567,7 @@ class Tests(unittest.TestCase):
'''))
ui = MockUI()
self.assertEqual(_get_ignore_from_manifest(filename, ui),
IgnoreList(['test.dat']))
IgnoreList().exclude('test.dat'))
self.assertEqual(ui.warnings, [
"%s, line 2: continuation line immediately precedes end-of-file" % filename,
])
@ -726,18 +686,16 @@ class TestConfiguration(unittest.TestCase):
with open('setup.cfg', 'w') as f:
f.write('[check-manifest]\nignore = foo\n bar*\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore,
check_manifest.IgnoreList.default()
+ check_manifest.IgnoreList(['foo', 'bar*']))
expected = check_manifest.IgnoreList.default().global_exclude('foo', 'bar*')
self.assertEqual(ignore, expected)
def test_read_pyproject_config_extra_ignores(self):
import check_manifest
with open('pyproject.toml', 'w') as f:
f.write('[tool.check-manifest]\nignore = ["foo", "bar*"]\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore,
check_manifest.IgnoreList.default()
+ check_manifest.IgnoreList(['foo', 'bar*']))
expected = check_manifest.IgnoreList.default().global_exclude('foo', 'bar*')
self.assertEqual(ignore, expected)
def test_read_setup_config_override_ignores(self):
import check_manifest
@ -745,7 +703,8 @@ class TestConfiguration(unittest.TestCase):
f.write('[check-manifest]\nignore = foo\n\n bar\n')
f.write('ignore-default-rules = yes\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore, check_manifest.IgnoreList(['foo', 'bar']))
expected = check_manifest.IgnoreList().global_exclude('foo', 'bar')
self.assertEqual(ignore, expected)
def test_read_pyproject_config_override_ignores(self):
import check_manifest
@ -753,7 +712,8 @@ class TestConfiguration(unittest.TestCase):
f.write('[tool.check-manifest]\nignore = ["foo", "bar"]\n')
f.write('ignore-default-rules = true\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore, check_manifest.IgnoreList(['foo', 'bar']))
expected = check_manifest.IgnoreList().global_exclude('foo', 'bar')
self.assertEqual(ignore, expected)
def test_read_setup_config_ignore_bad_ideas(self):
import check_manifest
@ -763,8 +723,8 @@ class TestConfiguration(unittest.TestCase):
' foo\n'
' bar*\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore_bad_ideas,
check_manifest.IgnoreList(['foo', 'bar*']))
expected = check_manifest.IgnoreList().global_exclude('foo', 'bar*')
self.assertEqual(ignore_bad_ideas, expected)
def test_read_pyproject_config_ignore_bad_ideas(self):
import check_manifest
@ -772,8 +732,8 @@ class TestConfiguration(unittest.TestCase):
f.write('[tool.check-manifest]\n'
'ignore-bad-ideas = ["foo", "bar*"]\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore_bad_ideas,
check_manifest.IgnoreList(['foo', 'bar*']))
expected = check_manifest.IgnoreList().global_exclude('foo', 'bar*')
self.assertEqual(ignore_bad_ideas, expected)
def test_read_manifest_no_manifest(self):
import check_manifest
@ -782,12 +742,12 @@ class TestConfiguration(unittest.TestCase):
def test_read_manifest(self):
import check_manifest
from check_manifest import _glob_to_regexp as g2r, IgnoreList
from check_manifest import IgnoreList
with open('MANIFEST.in', 'w') as f:
f.write('exclude *.gif\n')
f.write('global-exclude *.png\n')
ignore = check_manifest.read_manifest(self.ui)
self.assertEqual(ignore, IgnoreList(['*.png'], [g2r('*.gif')]))
self.assertEqual(ignore, IgnoreList().exclude('*.gif').global_exclude('*.png'))
class TestMain(unittest.TestCase):
@ -835,15 +795,17 @@ class TestMain(unittest.TestCase):
import check_manifest
sys.argv.append('--ignore=x,y,z*')
check_manifest.main()
ignore = check_manifest.IgnoreList().global_exclude('x', 'y', 'z*')
self.assertEqual(self._check_manifest.call_args.kwargs['extra_ignore'],
check_manifest.IgnoreList(['x', 'y', 'z*']))
ignore)
def test_ignore_bad_ideas_args(self):
import check_manifest
sys.argv.append('--ignore-bad-ideas=x,y,z*')
check_manifest.main()
ignore = check_manifest.IgnoreList().global_exclude('x', 'y', 'z*')
self.assertEqual(self._check_manifest.call_args.kwargs['extra_ignore_bad_ideas'],
check_manifest.IgnoreList(['x', 'y', 'z*']))
ignore)
def test_verbose_arg(self):
import check_manifest
@ -1033,20 +995,16 @@ class VCSMixin(object):
self._create_and_add_to_vcs(['a.txt', 'b/b.txt', 'b/c/d.txt'])
self._commit()
self._create_files(['b/x.txt', 'd/d.txt', 'i.txt'])
j = os.path.join
self.assertEqual(get_vcs_files(self.ui),
['a.txt', 'b', j('b', 'b.txt'), j('b', 'c'),
j('b', 'c', 'd.txt')])
['a.txt', 'b/b.txt', 'b/c/d.txt'])
def test_get_vcs_files_added_but_uncommitted(self):
from check_manifest import get_vcs_files
self._init_vcs()
self._create_and_add_to_vcs(['a.txt', 'b/b.txt', 'b/c/d.txt'])
self._create_files(['b/x.txt', 'd/d.txt', 'i.txt'])
j = os.path.join
self.assertEqual(get_vcs_files(self.ui),
['a.txt', 'b', j('b', 'b.txt'), j('b', 'c'),
j('b', 'c', 'd.txt')])
['a.txt', 'b/b.txt', 'b/c/d.txt'])
def test_get_vcs_files_deleted_but_not_removed(self):
if self.vcs.command == 'bzr':
@ -1066,8 +1024,7 @@ class VCSMixin(object):
self._commit()
self._create_files(['b/x.txt', 'd/d.txt', 'i.txt'])
os.chdir('b')
j = os.path.join
self.assertEqual(get_vcs_files(self.ui), ['b.txt', 'c', j('c', 'd.txt')])
self.assertEqual(get_vcs_files(self.ui), ['b.txt', 'c/d.txt'])
def test_get_vcs_files_nonascii_filenames(self):
# This test will fail if your locale is incapable of expressing
@ -1145,20 +1102,16 @@ class TestGit(VCSMixin, unittest.TestCase):
self.vcs._run('git', 'submodule', 'update', '--init', '--recursive')
self.assertEqual(
get_vcs_files(self.ui),
[fn.replace('/', os.path.sep) for fn in [
[
'.gitmodules',
'file5',
'sub1',
'sub1/file1',
'sub1/file2',
'subdir',
'subdir/file6',
'subdir/sub2',
'subdir/sub2/.gitmodules',
'subdir/sub2/file3',
'subdir/sub2/sub3',
'subdir/sub2/sub3/file4',
]])
])
class BzrHelper(VCSHelper):
@ -1251,9 +1204,14 @@ class SvnHelper(VCSHelper):
self._run('svn', 'co', 'file:///' + os.path.abspath('repo').replace(os.path.sep, '/'), 'checkout')
os.chdir('checkout')
def _add_directories_and_sort(self, filelist):
from check_manifest import normalize_names
names = set(normalize_names(filelist))
names.update([posixpath.dirname(fn) for fn in names])
return sorted(names - {''})
def _add_to_vcs(self, filenames):
from check_manifest import add_directories_and_sort
self._run('svn', 'add', '-N', '--', *add_directories_and_sort(filenames))
self._run('svn', 'add', '-N', '--', *self._add_directories_and_sort(filenames))
def _commit(self):
self._run('svn', 'commit', '-m', 'Initial')
@ -1271,9 +1229,8 @@ class TestSvn(VCSMixin, unittest.TestCase):
self.vcs._run('svn', 'up')
self._create_files(['a.txt', 'ext/b.txt'])
self.vcs._run('svn', 'add', 'a.txt', 'ext/b.txt')
j = os.path.join
self.assertEqual(get_vcs_files(self.ui),
['a.txt', 'ext', j('ext', 'b.txt')])
['a.txt', 'ext/b.txt'])
class TestSvnExtraErrors(unittest.TestCase):
@ -1394,15 +1351,182 @@ class TestUserInterface(unittest.TestCase):
class TestIgnoreList(unittest.TestCase):
def setUp(self):
from check_manifest import IgnoreList
self.ignore = IgnoreList()
def test_repr(self):
from check_manifest import IgnoreList
ignore = IgnoreList(['a'], ['b'])
self.assertEqual(repr(ignore), "IgnoreList(['a'], ['b'])")
ignore = IgnoreList()
self.assertEqual(repr(ignore), "IgnoreList([])")
def test_exclude_pattern(self):
self.ignore.exclude('*.txt')
self.assertEqual(self.ignore.filter([
'foo.md',
'bar.txt',
'subdir/bar.txt',
]), [
'foo.md',
'subdir/bar.txt',
])
def test_failed_addition(self):
def test_exclude_file(self):
self.ignore.exclude('bar.txt')
self.assertEqual(self.ignore.filter([
'foo.md',
'bar.txt',
'subdir/bar.txt',
]), [
'foo.md',
'subdir/bar.txt',
])
def test_exclude_doest_apply_to_directories(self):
self.ignore.exclude('subdir')
self.assertEqual(self.ignore.filter([
'foo.md',
'subdir/bar.txt',
]), [
'foo.md',
'subdir/bar.txt',
])
def test_global_exclude(self):
self.ignore.global_exclude('a*.txt')
self.assertEqual(self.ignore.filter([
'bar.txt', # make sure full filenames are matched
'afile.txt',
'subdir/afile.txt',
'adir/file.txt', # make sure * doesn't match /
]), [
# no bar.txt! because https://bugs.python.org/issue14106
'adir/file.txt',
])
def test_global_exclude_does_not_apply_to_directories(self):
self.ignore.global_exclude('subdir')
self.assertEqual(self.ignore.filter([
'bar.txt',
'subdir/afile.txt',
]), [
'bar.txt',
'subdir/afile.txt',
])
def test_recursive_exclude(self):
self.ignore.recursive_exclude('subdir', 'a*.txt')
self.assertEqual(self.ignore.filter([
'afile.txt',
'subdir/afile.txt',
'subdir/extra/afile.txt',
'subdir/adir/file.txt',
'other/afile.txt',
]), [
'afile.txt',
'subdir/adir/file.txt',
'other/afile.txt',
])
def test_recursive_exclude_does_not_apply_to_directories(self):
self.ignore.recursive_exclude('subdir', 'dir')
self.assertEqual(self.ignore.filter([
'afile.txt',
'subdir/dir/afile.txt',
]), [
'afile.txt',
'subdir/dir/afile.txt',
])
def test_recursive_exclude_can_prune(self):
self.ignore.recursive_exclude('subdir', '*')
self.assertEqual(self.ignore.filter([
'afile.txt',
'subdir/afile.txt',
'subdir/dir/afile.txt',
'subdir/dir/dir/afile.txt',
]), [
'afile.txt',
])
def test_prune(self):
self.ignore.prune('subdir')
self.assertEqual(self.ignore.filter([
'foo.md',
'subdir/bar.txt',
'unrelated/subdir/baz.txt',
]), [
'foo.md',
'unrelated/subdir/baz.txt',
])
def test_prune_subdir(self):
self.ignore.prune('a/b')
self.assertEqual(self.ignore.filter([
'foo.md',
'a/b/bar.txt',
'a/c/bar.txt',
]), [
'foo.md',
'a/c/bar.txt',
])
def test_prune_glob(self):
self.ignore.prune('su*r')
self.assertEqual(self.ignore.filter([
'foo.md',
'subdir/bar.txt',
'unrelated/subdir/baz.txt',
]), [
'foo.md',
'unrelated/subdir/baz.txt',
])
def test_prune_glob_is_not_too_greedy(self):
self.ignore.prune('su*r')
self.assertEqual(self.ignore.filter([
'foo.md',
# super-unrelated/subdir matches su*r if you allow * to match /,
# which fnmatch does!
'super-unrelated/subdir/qux.txt',
]), [
'foo.md',
'super-unrelated/subdir/qux.txt',
])
def test_default_excludes_pkg_info(self):
from check_manifest import IgnoreList
ignore = IgnoreList.default()
self.assertEqual(ignore.filter([
'PKG-INFO',
'bar.txt',
]), [
'bar.txt',
])
def test_default_excludes_egg_info(self):
from check_manifest import IgnoreList
ignore = IgnoreList.default()
self.assertEqual(ignore.filter([
'mypackage.egg-info/PKG-INFO',
'mypackage.egg-info/SOURCES.txt',
'mypackage.egg-info/requires.txt',
'bar.txt',
]), [
'bar.txt',
])
def test_default_excludes_egg_info_in_a_subdirectory(self):
from check_manifest import IgnoreList
with self.assertRaises(TypeError):
IgnoreList() + 42
ignore = IgnoreList.default()
self.assertEqual(ignore.filter([
'src/mypackage.egg-info/PKG-INFO',
'src/mypackage.egg-info/SOURCES.txt',
'src/mypackage.egg-info/requires.txt',
'bar.txt',
]), [
'bar.txt',
])
def pick_installed_vcs():
@ -1520,7 +1644,8 @@ class TestCheckManifest(unittest.TestCase):
from check_manifest import check_manifest, IgnoreList
self._create_repo_with_code()
self._add_to_vcs('unrelated.txt')
self.assertTrue(check_manifest(extra_ignore=IgnoreList(['*.txt'])),
ignore = IgnoreList().global_exclude('*.txt')
self.assertTrue(check_manifest(extra_ignore=ignore),
sys.stderr.getvalue())
def test_suggestions(self):
@ -1630,7 +1755,8 @@ class TestCheckManifest(unittest.TestCase):
self._add_to_vcs('foo.egg-info')
self._add_to_vcs('moo.mo')
self._add_to_vcs(os.path.join('subdir', 'bar.egg-info'))
self.assertFalse(check_manifest(extra_ignore_bad_ideas=IgnoreList(['*.mo'])))
ignore = IgnoreList().global_exclude('*.mo')
self.assertFalse(check_manifest(extra_ignore_bad_ideas=ignore))
self.assertIn("you have foo.egg-info in source control!",
sys.stderr.getvalue())
self.assertNotIn("moo.mo", sys.stderr.getvalue())

2
tox.ini

@ -1,6 +1,6 @@
[tox]
envlist =
py27,py35,py36,py37,py38,pypy,pypy3,flake8
py35,py36,py37,py38,pypy3,flake8
[testenv]
passenv = LANG LC_CTYPE LC_ALL MSYSTEM

Loading…
Cancel
Save