mirrors-python
/
check-manifest
mirror of https://github.com/mgedmin/check-manifest.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity Software
Browse Source

Rewrite the IgnoreList logic to be more distutils-like 

This should eliminate most of the inconsistencies with how distutils
treats MANIFEST.in files.  (Some inconsistencies remain: distutils
allows you to add back previously excluded files.)

The way I went about it:

- explicitly defined the internal canonical format for lists of file
  names

- changed it to use / instead of os.path.sep

- changed it to omit directories and list only files, which makes ignore
  handling simpler

To elaborate on the last point, here are some examples:

1. 'exclude foo' in MANIFEST.in works only on regular files,
   not directories, but how can we know if all we have is a string?

2. if you exclude all the files in a directory, distutils will omit that
   directory from the sdist, but it's difficult to do the same when all
   you have is a list of strings

Next, I changed IgnoreList to avoid fnmatch and be entirely based on
regular expressions, and this time I'm using distutils's own
translate_pattern() to produce exactly the same regular expressions
that distutils itself will use for matching.

This may introduce some surprising behaviour, like 'global-exclude a*.txt'
now matches on suffixes of file names, so it would exclude files named
bar.txt!  See https://bugs.python.org/issue14106.

Then tests started failing on Python 2.7, but all I get from the
__repr__ is a bunch of opaque <_sre.SRE_Pattern object at 0xdeadbeef>,
which are impossible to debug.  Therefore I'm declaring Python 2.7
support officially dropped!

check-manifests's own configuration (--ignore or [check-manifest]
ignore=) is now treated as a global-exclude statement in MANIFEST.in,
which means:

- it's matched anywhere in the file tree
- it's matched against a filename suffix due to the above-mentioned
  Python bug
- it's ignored if it matches a directory

You can ignore directories only by ignoring every file inside it.
There's no way currently of specifying that that works for arbitrarily
nested trees, but you can try --ignore=dir/*,dir/*/*,dir/*/*/*
until you get the result you want.

This decision is not cast in stone: I may in the future change the
handling of --ignore to match files and directories, because there's no
reason it has to be distutils-compatible.

Expect regressions on Windows.  I'll fix any that I find.
pull/117/head
Marius Gedminas 2 years ago
parent
cce344006b
commit
f56a51f2b2
8 changed files with 387 additions and 287 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 1
      .gitignore
  2. 2
      .travis.yml
  3. 22
      CHANGES.rst
  4. 1
      appveyor.yml
  5. 238
      check_manifest.py
  6. 4
      setup.py
  7. 404
      tests.py
  8. 2
      tox.ini

1
.gitignore vendored
Unescape View File

@ -10,3 +10,4 @@ tmp/
build/
tags
coverage.xml
.mypy_cache/

2
.travis.yml
Unescape View File

@ -1,12 +1,10 @@
language: python
cache: pip
python:
- 2.7
- 3.5
- 3.6
- 3.7
- 3.8
- pypy
- pypy3
env:
- FORCE_TEST_VCS=bzr

22
CHANGES.rst
Unescape View File

@ -8,6 +8,28 @@ Changelog
- Added ``-q``/``--quiet`` command line argument. This will reduce the verbosity
of informational output, e.g. for use in a CI pipeline.
- Rewrote the ignore logic to be more compatible with distutils. This might
have introduced some regressions, so please file bugs! One side effect of
this is that ``--ignore`` (or the ``ignore`` setting in the config file)
is now handled the same way as ``global-exclude`` in a ``MANIFEST.in``, which
means:
- it's matched anywhere in the file tree
- it's matched against a filename *suffix* rather than an entire filename,
due to https://bugs.python.org/issue14106
- it's ignored if it matches a directory
You can ignore directories only by ignoring every file inside it.
There's no way currently of specifying that that works for arbitrarily
nested trees, but you can try ``--ignore=dir/*,dir/*/*,dir/*/*/*``
until you get the result you want.
This decision is not cast in stone: I may in the future change the
handling of ``--ignore`` to match files and directories, because there's no
reason it has to be distutils-compatible.
- Drop Python 2.7 support.
0.41 (2020-02-25)
-----------------

1
appveyor.yml
Unescape View File

@ -4,7 +4,6 @@ environment:
matrix:
# https://www.appveyor.com/docs/installed-software#python lists available
# versions
- PYTHON: "C:\\Python27"
- PYTHON: "C:\\Python35"
- PYTHON: "C:\\Python36"
- PYTHON: "C:\\Python37"

238
check_manifest.py
Unescape View File

@ -30,8 +30,8 @@ import tarfile
import tempfile
import unicodedata
import zipfile
from distutils.filelist import glob_to_re
from contextlib import contextmanager, closing
from distutils.filelist import translate_pattern
from distutils.text_file import TextFile
try:
@ -266,16 +266,39 @@ def get_one_file_in(dirname):
#
# - contain Unicode filenames (normalized to NFC on OS X)
# - be sorted
# - use os.path.sep as the directory separator
# (I'm not sure this is done correctly at the moment!)
# - list directories as well as files (but without trailing slashes)
# - use / as the directory separator
# - list only files, but not directories
#
# We get these file lists from various sources (zip files, tar files, version
# control systems) and we have to normalize them into our common format before
# comparing
# comparing.
#
def canonical_file_list(filelist):
"""Return the file list convered to a canonical form.
This means:
- converted to Unicode normal form C, when running on Mac OS X
- sorted alphabetically
- use / as the directory separator
- list files but not directories
Caveat: since it works on file lists taken from archives and such, it
doesn't know whether a particular filename refers to a file or a directory,
unless it finds annother filename that is inside the first one. In other
words, canonical_file_list() will not remove the names of empty directories
if those appear in the initial file list.
"""
names = set(normalize_names(filelist))
for name in list(names):
while name:
name = posixpath.dirname(name)
names.discard(name)
return sorted(names)
def get_sdist_file_list(sdist_filename, ignore):
"""Return the list of interesting files in a source distribution.
@ -284,27 +307,9 @@ def get_sdist_file_list(sdist_filename, ignore):
Supports .tar.gz and .zip sdists.
"""
# The sorted() is probably redundant here, given that
# get_archive_file_list() returns a sorted list, but I'm not 100% sure
# normalize_names() won't affect the sort order.
# TODO: move normalize_names() into get_archive_file_list(), because
# get_vcs_files() first normalizes and then calls
# add_directories_and_sort().
return sorted(normalize_names(strip_sdist_extras(ignore,
strip_toplevel_name(get_archive_file_list(sdist_filename)))))
def unicodify(filename):
"""Make sure filename is Unicode.
Because the tarfile module on Python 2 doesn't return Unicode.
"""
if isinstance(filename, bytes):
# XXX: Ah, but is it right to use the locale encoding here, or should I
# use sys.getfilesystemencoding()? A good question!
return filename.decode(locale.getpreferredencoding())
else:
return filename
return strip_sdist_extras(
ignore,
strip_toplevel_name(get_archive_file_list(sdist_filename)))
def get_archive_file_list(archive_filename):
@ -321,10 +326,20 @@ def get_archive_file_list(archive_filename):
else:
raise Failure('Unrecognized archive type: %s'
% os.path.basename(archive_filename))
# Hm, should we normalize_names() here? Maybe, maybe not. The only caller
# of get_archive_file_list() is get_sdist_file_list(), and it calls the
# normalize_names().
return add_directories_and_sort(filelist)
return canonical_file_list(filelist)
def unicodify(filename):
"""Make sure filename is Unicode.
Because the tarfile module on Python 2 doesn't return Unicode.
"""
if isinstance(filename, bytes):
# XXX: Ah, but is it right to use the locale encoding here, or should I
# use sys.getfilesystemencoding()? A good question!
return filename.decode(locale.getpreferredencoding())
else:
return filename
def strip_toplevel_name(filelist):
@ -360,8 +375,7 @@ def strip_toplevel_name(filelist):
def add_prefix_to_each(prefix, filelist):
"""Add a prefix to each name in a file list.
>>> filelist = add_prefix_to_each('foo/bar', ['a', 'b', 'c/d'])
>>> [f.replace(os.path.sep, '/') for f in filelist]
>>> add_prefix_to_each('foo/bar', ['a', 'b', 'c/d'])
['foo/bar/a', 'foo/bar/b', 'foo/bar/c/d']
"""
@ -539,12 +553,12 @@ def detect_vcs(ui):
def get_vcs_files(ui):
"""List all files under version control in the current directory."""
vcs = detect_vcs(ui)
return add_directories_and_sort(normalize_names(vcs.get_versioned_files()))
return canonical_file_list(vcs.get_versioned_files())
def normalize_names(names):
"""Normalize file names."""
return list(map(normalize_name, names))
return [normalize_name(name) for name in names]
def normalize_name(name):
@ -555,8 +569,8 @@ def normalize_name(name):
And encodings may trip us up too, especially when comparing lists
of files. Plus maybe lowercase versus uppercase.
"""
name = os.path.normpath(name)
name = unicodify(name)
name = os.path.normpath(name).replace(os.path.sep, '/')
name = unicodify(name) # XXX is this necessary?
if sys.platform == 'darwin':
# Mac OS X may have problems comparing non-ASCII filenames, so
# we convert them.
@ -564,120 +578,77 @@ def normalize_name(name):
return name
def add_directories_and_sort(names):
"""Git/Mercurial/zip files omit directories, let's add them back."""
# Let's make sure we iterate names once, in case it's an iterator and not a
# list.
res = list(names)
seen = set(res)
for name in list(res):
while True:
name = os.path.dirname(name)
if not name or name in seen:
break
res.append(name)
seen.add(name)
return sorted(res)
#
# Packaging logic
#
class IgnoreList(object):
def __init__(self, ignore=(), _ignore_regexps=()):
self.ignore = list(ignore)
self.ignore_regexps = list(_ignore_regexps)
def __init__(self):
self._regexps = []
@classmethod
def default(cls):
return cls(DEFAULT_IGNORE)
return (
cls()
# these are always generated
.global_exclude('PKG-INFO')
.global_exclude('*.egg-info/*')
# setup.cfg is always generated, but sometimes also kept in source control
.global_exclude('setup.cfg')
# it's not a problem if the sdist is lacking these files:
.global_exclude(
'.hgtags', '.hgsigs', '.hgignore', '.gitignore', '.bzrignore',
'.gitattributes',
)
# GitHub template files
.prune('.github')
# we can do without these in sdists
.global_exclude('.travis.yml')
.global_exclude('Jenkinsfile')
# It's convenient to ship compiled .mo files in sdists, but they
# shouldn't be checked in, so don't complain that they're missing
# from VCS
.global_exclude('*.mo')
)
def clear(self):
self.ignore = []
self.ignore_regexps = []
self._regexps = []
def __repr__(self):
return 'IgnoreList(%r, %r)' % (self.ignore, self.ignore_regexps)
return 'IgnoreList(%r)' % (self._regexps)
def __eq__(self, other):
return (isinstance(other, IgnoreList) and self.ignore == other.ignore
and self.ignore_regexps == other.ignore_regexps)
return isinstance(other, IgnoreList) and self._regexps == other._regexps
def __iadd__(self, other):
assert isinstance(other, IgnoreList)
self.ignore += other.ignore
self.ignore_regexps += other.ignore_regexps
self._regexps += other._regexps
return self
def __add__(self, other):
if not isinstance(other, IgnoreList):
return NotImplemented
result = IgnoreList()
result.ignore = self.ignore + other.ignore
result.ignore_regexps = self.ignore_regexps + other.ignore_regexps
return result
def exclude(self, *patterns):
# An exclude of 'dirname/*css' can match 'dirname/foo.css'
# but not 'dirname/subdir/bar.css'. We need a regular
# expression for that, since fnmatch doesn't pay attention to
# directory separators.
for pat in patterns:
if '*' in pat or '?' in pat or '[!' in pat:
self.ignore_regexps.append(_glob_to_regexp(pat))
else:
self.ignore.append(pat)
self._regexps.append(translate_pattern(pat, anchor=True))
return self
def global_exclude(self, *patterns):
for pat in patterns:
self.ignore.append(pat)
self._regexps.append(translate_pattern(pat, anchor=False))
return self
def recursive_exclude(self, dirname, *patterns):
# Strip path separator for clarity.
dirname = dirname.rstrip('/\\')
for pattern in patterns:
if pattern.startswith('*'):
self.ignore.append(dirname + os.path.sep + pattern)
else:
# 'recursive-exclude plone metadata.xml' should
# exclude plone/metadata.xml and
# plone/*/metadata.xml, where * can be any number
# of sub directories. We could use a regexp, but
# two ignores seems easier.
self.ignore.append(dirname + os.path.sep + pattern)
self.ignore.append(
dirname + os.path.sep + '*' + os.path.sep + pattern)
for pat in patterns:
self._regexps.append(translate_pattern(pat, prefix=dirname))
return self
def prune(self, subdir):
subdir = subdir.rstrip('/\\')
self.ignore.append(subdir)
self.ignore.append(subdir + os.path.sep + '*')
self._regexps.append(translate_pattern(None, prefix=subdir))
return self
def filter(self, filelist):
return [name for name in filelist
if not file_matches(name, self.ignore)
and not file_matches_regexps(name, self.ignore_regexps)]
# it's fine if any of these are missing in the VCS or in the sdist
DEFAULT_IGNORE = [
'PKG-INFO', # always generated
'*.egg-info', # always generated
'*.egg-info/*', # always generated
'setup.cfg', # always generated, sometimes also kept in source control
# it's not a problem if the sdist is lacking these files:
'.hgtags', '.hgsigs', '.hgignore', '.gitignore', '.bzrignore',
'.gitattributes',
'.github', # GitHub template files
'.github/*', # GitHub template files
'.travis.yml',
'Jenkinsfile',
# it's convenient to ship compiled .mo files in sdists, but they shouldn't
# be checked in
'*.mo',
]
if not any(rx.search(name) for rx in self._regexps)]
WARN_ABOUT_FILES_IN_VCS = [
# generated files should not be committed into the VCS
@ -692,9 +663,7 @@ WARN_ABOUT_FILES_IN_VCS = [
'.#*',
]
_sep = re.escape(os.path.sep)
SUGGESTIONS = [(re.compile(pattern.replace('/', _sep)), suggestion) for pattern, suggestion in [
SUGGESTIONS = [(re.compile(pattern), suggestion) for pattern, suggestion in [
# regexp -> suggestion
('^([^/]+[.](cfg|ini))$', r'include \1'),
('^([.]travis[.]yml)$', r'include \1'),
@ -793,21 +762,10 @@ def read_manifest(ui):
return _get_ignore_from_manifest('MANIFEST.in', ui)
def _glob_to_regexp(pat):
"""Compile a glob pattern into a regexp.
We need to do this because fnmatch allows * to match /, which we
don't want. E.g. a MANIFEST.in exclude of 'dirname/*css' should
match 'dirname/foo.css' but not 'dirname/subdir/bar.css'.
"""
return glob_to_re(pat)
def _get_ignore_from_manifest(filename, ui):
"""Gather the various ignore patterns from a MANIFEST.in.
Returns a list of standard ignore patterns and a list of regular
expressions to ignore.
Returns an IgnoreList instance.
"""
class MyTextFile(TextFile):
@ -838,8 +796,7 @@ def _get_ignore_from_manifest_lines(lines, ui):
'lines' should be a list of strings with comments removed
and continuation lines joined.
Returns a list of standard ignore patterns and a list of regular
expressions to ignore.
Returns an IgnoreList instance.
"""
ignore = IgnoreList()
for line in lines:
@ -890,11 +847,6 @@ def file_matches(filename, patterns):
for pat in patterns)
def file_matches_regexps(filename, patterns):
"""Does this filename match any of the regular expressions?"""
return any(re.match(pat, filename) for pat in patterns)
def strip_sdist_extras(ignore, filelist):
"""Strip generated files that are only present in source distributions.
@ -911,7 +863,11 @@ def find_bad_ideas(filelist):
def find_suggestions(filelist):
"""Suggest MANIFEST.in patterns for missing files."""
"""Suggest MANIFEST.in patterns for missing files.
Returns two lists: one with suggested MANIGEST.in commands, and one with
files for which no suggestions were offered.
"""
suggestions = set()
unknowns = []
for filename in filelist:

4
setup.py
Unescape View File

@ -42,8 +42,6 @@ setup(
'License :: OSI Approved :: MIT License',
'Operating System :: OS Independent',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
@ -56,7 +54,7 @@ setup(
py_modules=['check_manifest'],
zip_safe=False,
python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
python_requires=">=3.5",
install_requires=['toml', 'pep517'],
extras_require={
'test': ['mock'],

404
tests.py
Unescape View File

@ -4,6 +4,7 @@ import codecs
import doctest
import locale
import os
import posixpath
import shutil
import subprocess
import sys
@ -219,7 +220,7 @@ class Tests(unittest.TestCase):
filename = os.path.join(self.make_temp_dir(), 'archive.zip')
self.create_zip_file(filename, ['a', 'b/c'])
self.assertEqual(get_archive_file_list(filename),
['a', 'b', 'b/c'])
['a', 'b/c'])
def test_get_archive_file_list_zip_nonascii(self):
from check_manifest import get_archive_file_list
@ -234,7 +235,7 @@ class Tests(unittest.TestCase):
filename = os.path.join(self.make_temp_dir(), 'archive.tar')
self.create_tar_file(filename, ['a', 'b/c'])
self.assertEqual(get_archive_file_list(filename),
['a', 'b', 'b/c'])
['a', 'b/c'])
def test_get_archive_file_list_tar_nonascii(self):
from check_manifest import get_archive_file_list
@ -287,32 +288,29 @@ class Tests(unittest.TestCase):
self.assertEqual(normalize_names(["a", j("b", ""), j("c", "d"),
j("e", "f", ""),
j("g", "h", "..", "i")]),
["a", "b", j("c", "d"), j("e", "f"), j("g", "i")])
["a", "b", "c/d", "e/f", "g/i"])
def test_add_directories_and_sort(self):
from check_manifest import add_directories_and_sort
def test_canonical_file_list(self):
from check_manifest import canonical_file_list
j = os.path.join
self.assertEqual(
add_directories_and_sort(['a', 'b', j('c', 'd'), j('e', 'f')]),
['a', 'b', 'c', j('c', 'd'), 'e', j('e', 'f')])
canonical_file_list(['b', 'a', 'c', j('c', 'd'), j('e', 'f'),
'g', j('g', 'h', 'i', 'j')]),
['a', 'b', 'c/d', 'e/f', 'g/h/i/j'])
def test_file_matches(self):
from check_manifest import file_matches
# On Windows we might get the pattern list from setup.cfg using / as
# the directory separator, but the filenames we're matching against
# will use os.path.sep
patterns = ['setup.cfg', '*.egg-info', '*.egg-info/*']
j = os.path.join
self.assertFalse(file_matches('setup.py', patterns))
self.assertTrue(file_matches('setup.cfg', patterns))
self.assertTrue(file_matches(j('src', 'zope.foo.egg-info'), patterns))
self.assertTrue(
file_matches(j('src', 'zope.foo.egg-info', 'SOURCES.txt'),
patterns))
self.assertTrue(file_matches('src/zope.foo.egg-info', patterns))
self.assertTrue(file_matches('src/zope.foo.egg-info/SOURCES.txt',
patterns))
def test_strip_sdist_extras(self):
from check_manifest import strip_sdist_extras, IgnoreList
filelist = list(map(os.path.normpath, [
from check_manifest import canonical_file_list
filelist = canonical_file_list([
'.github',
'.github/ISSUE_TEMPLATE',
'.github/ISSUE_TEMPLATE/bug_report.md',
@ -331,8 +329,8 @@ class Tests(unittest.TestCase):
'src/zope/foo/language.mo',
'src/zope.foo.egg-info',
'src/zope.foo.egg-info/SOURCES.txt',
]))
expected = list(map(os.path.normpath, [
])
expected = canonical_file_list([
'setup.py',
'README.txt',
'src',
@ -341,12 +339,13 @@ class Tests(unittest.TestCase):
'src/zope/foo',
'src/zope/foo/__init__.py',
'src/zope/foo/language.po',
]))
])
ignore = IgnoreList.default()
self.assertEqual(strip_sdist_extras(ignore, filelist), expected)
def test_strip_sdist_extras_with_manifest(self):
from check_manifest import strip_sdist_extras, IgnoreList
from check_manifest import canonical_file_list
from check_manifest import _get_ignore_from_manifest_lines
manifest_in = textwrap.dedent("""
graft src
@ -355,7 +354,7 @@ class Tests(unittest.TestCase):
prune src/dump
recursive-exclude src/zope *.sh
""")
filelist = list(map(os.path.normpath, [
filelist = canonical_file_list([
'.github/ISSUE_TEMPLATE/bug_report.md',
'.gitignore',
'setup.py',
@ -377,8 +376,8 @@ class Tests(unittest.TestCase):
'src/zope/foo/foohelper.sh',
'src/zope.foo.egg-info',
'src/zope.foo.egg-info/SOURCES.txt',
]))
expected = list(map(os.path.normpath, [
])
expected = canonical_file_list([
'setup.py',
'MANIFEST.in',
'README.txt',
@ -390,7 +389,7 @@ class Tests(unittest.TestCase):
'src/zope/foo/__init__.py',
'src/zope/foo/language.po',
'src/zope/foo/config.cfg',
]))
])
ignore = IgnoreList.default()
ignore += _get_ignore_from_manifest_lines(manifest_in.splitlines(), self.ui)
result = strip_sdist_extras(ignore, filelist)
@ -474,80 +473,39 @@ class Tests(unittest.TestCase):
self.assertEqual(e('dist/foo_bar-1.2.3.dev4+g12345.zip'), '1.2.3.dev4+g12345')
self.assertEqual(e('dist/foo_bar-1.2.3.dev4+g12345.tar.gz'), '1.2.3.dev4+g12345')
def test_glob_to_regexp(self):
from check_manifest import _glob_to_regexp as g2r
sep = os.path.sep.replace('\\', '\\\\')
if sys.version_info >= (3, 7):
self.assertEqual(g2r('foo.py'), r'(?s:foo\.py)\Z')
self.assertEqual(g2r('foo/bar.py'), r'(?s:foo/bar\.py)\Z')
self.assertEqual(g2r('foo*.py'), r'(?s:foo[^%s]*\.py)\Z' % sep)
self.assertEqual(g2r('foo?.py'), r'(?s:foo[^%s]\.py)\Z' % sep)
self.assertEqual(g2r('foo[123].py'), r'(?s:foo[123]\.py)\Z')
self.assertEqual(g2r('foo[!123].py'), r'(?s:foo[^123]\.py)\Z')
self.assertEqual(g2r('foo/*.py'), r'(?s:foo/[^%s]*\.py)\Z' % sep)
elif sys.version_info >= (3, 6):
self.assertEqual(g2r('foo.py'), r'(?s:foo\.py)\Z')
self.assertEqual(g2r('foo/bar.py'), r'(?s:foo\/bar\.py)\Z')
self.assertEqual(g2r('foo*.py'), r'(?s:foo[^%s]*\.py)\Z' % sep)
self.assertEqual(g2r('foo?.py'), r'(?s:foo[^%s]\.py)\Z' % sep)
self.assertEqual(g2r('foo[123].py'), r'(?s:foo[123]\.py)\Z')
self.assertEqual(g2r('foo[!123].py'), r'(?s:foo[^123]\.py)\Z')
self.assertEqual(g2r('foo/*.py'), r'(?s:foo\/[^%s]*\.py)\Z' % sep)
else:
self.assertEqual(g2r('foo.py'), r'foo\.py\Z(?ms)')
self.assertEqual(g2r('foo/bar.py'), r'foo\/bar\.py\Z(?ms)')
self.assertEqual(g2r('foo*.py'), r'foo[^%s]*\.py\Z(?ms)' % sep)
self.assertEqual(g2r('foo?.py'), r'foo[^%s]\.py\Z(?ms)' % sep)
self.assertEqual(g2r('foo[123].py'), r'foo[123]\.py\Z(?ms)')
self.assertEqual(g2r('foo[!123].py'), r'foo[^123]\.py\Z(?ms)')
self.assertEqual(g2r('foo/*.py'), r'foo\/[^%s]*\.py\Z(?ms)' % sep)
def test_get_ignore_from_manifest_lines(self):
from check_manifest import _get_ignore_from_manifest_lines
from check_manifest import _glob_to_regexp as g2r
from check_manifest import IgnoreList
j = os.path.join
parse = partial(_get_ignore_from_manifest_lines, ui=self.ui)
# The return value is a tuple with two lists:
# ([<list of filename ignores>], [<list of regular expressions>])
self.assertEqual(parse([]),
IgnoreList([], []))
IgnoreList())
self.assertEqual(parse(['', ' ']),
IgnoreList([], []))
IgnoreList())
self.assertEqual(parse(['exclude *.cfg']),
IgnoreList([], [g2r('*.cfg')]))
IgnoreList().exclude('*.cfg'))
self.assertEqual(parse(['exclude *.cfg']),
IgnoreList([], [g2r('*.cfg')]))
IgnoreList().exclude('*.cfg'))
self.assertEqual(parse(['\texclude\t*.cfg foo.* bar.txt']),
IgnoreList(['bar.txt'], [g2r('*.cfg'), g2r('foo.*')]))
IgnoreList().exclude('*.cfg', 'foo.*', 'bar.txt'))
self.assertEqual(parse(['exclude some/directory/*.cfg']),
IgnoreList([], [g2r('some/directory/*.cfg')]))
IgnoreList().exclude('some/directory/*.cfg'))
self.assertEqual(parse(['include *.cfg']),
IgnoreList([], []))
IgnoreList())
self.assertEqual(parse(['global-exclude *.pyc']),
IgnoreList(['*.pyc'], []))
IgnoreList().global_exclude('*.pyc'))
self.assertEqual(parse(['global-exclude *.pyc *.sh']),
IgnoreList(['*.pyc', '*.sh'], []))
IgnoreList().global_exclude('*.pyc', '*.sh'))
self.assertEqual(parse(['recursive-exclude dir *.pyc']),
IgnoreList([j('dir', '*.pyc')], []))
IgnoreList().recursive_exclude('dir', '*.pyc'))
self.assertEqual(parse(['recursive-exclude dir *.pyc foo*.sh']),
IgnoreList([j('dir', '*.pyc'), j('dir', 'foo*.sh'),
j('dir', '*', 'foo*.sh')], []))
IgnoreList().recursive_exclude('dir', '*.pyc', 'foo*.sh'))
self.assertEqual(parse(['recursive-exclude dir nopattern.xml']),
IgnoreList([j('dir', 'nopattern.xml'),
j('dir', '*', 'nopattern.xml')], []))
IgnoreList().recursive_exclude('dir', 'nopattern.xml'))
# We should not fail when a recursive-exclude line is wrong:
self.assertEqual(parse(['recursive-exclude dirwithoutpattern']),
IgnoreList([], []))
IgnoreList())
self.assertEqual(parse(['prune dir']),
IgnoreList(['dir', j('dir', '*')], []))
# You should not add a slash at the end of a prune, but let's
# not fail over it or end up with double slashes.
self.assertEqual(parse(['prune dir/']),
IgnoreList(['dir', j('dir', '*')], []))
# You should also not have a leading slash
self.assertEqual(parse(['prune /dir']),
IgnoreList(['/dir', j('/dir', '*')], []))
IgnoreList().prune('dir'))
# And a mongo test case of everything at the end
text = textwrap.dedent("""
exclude *.02
@ -563,24 +521,26 @@ class Tests(unittest.TestCase):
""").splitlines()
self.assertEqual(
parse(text),
IgnoreList([
'bar.txt',
'*.10',
'*.11',
'*.12',
'30',
j('30', '*'),
j('40', '*.41'),
j('42', '*.43'),
j('42', '44.*'),
j('42', '*', '44.*'),
], [
g2r('*.02'),
g2r('*.03'),
g2r('04.*'),
g2r('*.05'),
g2r('some/directory/*.cfg'),
]))
IgnoreList()
.exclude('*.02', '*.03', '04.*', 'bar.txt', '*.05', 'some/directory/*.cfg')
.global_exclude('*.10', '*.11', '*.12')
.prune('30')
.recursive_exclude('40', '*.41')
.recursive_exclude('42', '*.43', '44.*')
)
def test_get_ignore_from_manifest_lines_warns(self):
from check_manifest import _get_ignore_from_manifest_lines, IgnoreList
parse = partial(_get_ignore_from_manifest_lines, ui=self.ui)
text = textwrap.dedent("""
graft a/
recursive-include /b *.txt
""").splitlines()
self.assertEqual(parse(text), IgnoreList())
self.assertEqual(self.ui.warnings, [
'ERROR: Trailing slashes are not allowed in MANIFEST.in on Windows: a/',
'ERROR: Leading slashes are not allowed in MANIFEST.in on Windows: /b',
])
def test_get_ignore_from_manifest(self):
from check_manifest import _get_ignore_from_manifest, IgnoreList
@ -595,7 +555,7 @@ class Tests(unittest.TestCase):
'''))
ui = MockUI()
self.assertEqual(_get_ignore_from_manifest(filename, ui),
IgnoreList(['test.dat']))
IgnoreList().exclude('test.dat'))
self.assertEqual(ui.warnings, [])
def test_get_ignore_from_manifest_warnings(self):
@ -607,7 +567,7 @@ class Tests(unittest.TestCase):
'''))
ui = MockUI()
self.assertEqual(_get_ignore_from_manifest(filename, ui),
IgnoreList(['test.dat']))
IgnoreList().exclude('test.dat'))
self.assertEqual(ui.warnings, [
"%s, line 2: continuation line immediately precedes end-of-file" % filename,
])
@ -726,18 +686,16 @@ class TestConfiguration(unittest.TestCase):
with open('setup.cfg', 'w') as f:
f.write('[check-manifest]\nignore = foo\n bar*\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore,
check_manifest.IgnoreList.default()
+ check_manifest.IgnoreList(['foo', 'bar*']))
expected = check_manifest.IgnoreList.default().global_exclude('foo', 'bar*')
self.assertEqual(ignore, expected)
def test_read_pyproject_config_extra_ignores(self):
import check_manifest
with open('pyproject.toml', 'w') as f:
f.write('[tool.check-manifest]\nignore = ["foo", "bar*"]\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore,
check_manifest.IgnoreList.default()
+ check_manifest.IgnoreList(['foo', 'bar*']))
expected = check_manifest.IgnoreList.default().global_exclude('foo', 'bar*')
self.assertEqual(ignore, expected)
def test_read_setup_config_override_ignores(self):
import check_manifest
@ -745,7 +703,8 @@ class TestConfiguration(unittest.TestCase):
f.write('[check-manifest]\nignore = foo\n\n bar\n')
f.write('ignore-default-rules = yes\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore, check_manifest.IgnoreList(['foo', 'bar']))
expected = check_manifest.IgnoreList().global_exclude('foo', 'bar')
self.assertEqual(ignore, expected)
def test_read_pyproject_config_override_ignores(self):
import check_manifest
@ -753,7 +712,8 @@ class TestConfiguration(unittest.TestCase):
f.write('[tool.check-manifest]\nignore = ["foo", "bar"]\n')
f.write('ignore-default-rules = true\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore, check_manifest.IgnoreList(['foo', 'bar']))
expected = check_manifest.IgnoreList().global_exclude('foo', 'bar')
self.assertEqual(ignore, expected)
def test_read_setup_config_ignore_bad_ideas(self):
import check_manifest
@ -763,8 +723,8 @@ class TestConfiguration(unittest.TestCase):
' foo\n'
' bar*\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore_bad_ideas,
check_manifest.IgnoreList(['foo', 'bar*']))
expected = check_manifest.IgnoreList().global_exclude('foo', 'bar*')
self.assertEqual(ignore_bad_ideas, expected)
def test_read_pyproject_config_ignore_bad_ideas(self):
import check_manifest
@ -772,8 +732,8 @@ class TestConfiguration(unittest.TestCase):
f.write('[tool.check-manifest]\n'
'ignore-bad-ideas = ["foo", "bar*"]\n')
ignore, ignore_bad_ideas = check_manifest.read_config()
self.assertEqual(ignore_bad_ideas,
check_manifest.IgnoreList(['foo', 'bar*']))
expected = check_manifest.IgnoreList().global_exclude('foo', 'bar*')
self.assertEqual(ignore_bad_ideas, expected)
def test_read_manifest_no_manifest(self):
import check_manifest
@ -782,12 +742,12 @@ class TestConfiguration(unittest.TestCase):
def test_read_manifest(self):
import check_manifest
from check_manifest import _glob_to_regexp as g2r, IgnoreList
from check_manifest import IgnoreList
with open('MANIFEST.in', 'w') as f:
f.write('exclude *.gif\n')
f.write('global-exclude *.png\n')
ignore = check_manifest.read_manifest(self.ui)
self.assertEqual(ignore, IgnoreList(['*.png'], [g2r('*.gif')]))
self.assertEqual(ignore, IgnoreList().exclude('*.gif').global_exclude('*.png'))
class TestMain(unittest.TestCase):
@ -835,15 +795,17 @@ class TestMain(unittest.TestCase):
import check_manifest
sys.argv.append('--ignore=x,y,z*')
check_manifest.main()
ignore = check_manifest.IgnoreList().global_exclude('x', 'y', 'z*')
self.assertEqual(self._check_manifest.call_args.kwargs['extra_ignore'],
check_manifest.IgnoreList(['x', 'y', 'z*']))
ignore)
def test_ignore_bad_ideas_args(self):
import check_manifest
sys.argv.append('--ignore-bad-ideas=x,y,z*')
check_manifest.main()
ignore = check_manifest.IgnoreList().global_exclude('x', 'y', 'z*')
self.assertEqual(self._check_manifest.call_args.kwargs['extra_ignore_bad_ideas'],
check_manifest.IgnoreList(['x', 'y', 'z*']))
ignore)
def test_verbose_arg(self):
import check_manifest
@ -1033,20 +995,16 @@ class VCSMixin(object):
self._create_and_add_to_vcs(['a.txt', 'b/b.txt', 'b/c/d.txt'])
self._commit()
self._create_files(['b/x.txt', 'd/d.txt', 'i.txt'])
j = os.path.join
self.assertEqual(get_vcs_files(self.ui),
['a.txt', 'b', j('b', 'b.txt'), j('b', 'c'),
j('b', 'c', 'd.txt')])
['a.txt', 'b/b.txt', 'b/c/d.txt'])
def test_get_vcs_files_added_but_uncommitted(self):
from check_manifest import get_vcs_files
self._init_vcs()
self._create_and_add_to_vcs(['a.txt', 'b/b.txt', 'b/c/d.txt'])
self._create_files(['b/x.txt', 'd/d.txt', 'i.txt'])
j = os.path.join
self.assertEqual(get_vcs_files(self.ui),
['a.txt', 'b', j('b', 'b.txt'), j('b', 'c'),
j('b', 'c', 'd.txt')])
['a.txt', 'b/b.txt', 'b/c/d.txt'])
def test_get_vcs_files_deleted_but_not_removed(self):
if self.vcs.command == 'bzr':
@ -1066,8 +1024,7 @@ class VCSMixin(object):
self._commit()
self._create_files(['b/x.txt', 'd/d.txt', 'i.txt'])
os.chdir('b')
j = os.path.join
self.assertEqual(get_vcs_files(self.ui), ['b.txt', 'c', j('c', 'd.txt')])
self.assertEqual(get_vcs_files(self.ui), ['b.txt', 'c/d.txt'])
def test_get_vcs_files_nonascii_filenames(self):
# This test will fail if your locale is incapable of expressing
@ -1145,20 +1102,16 @@ class TestGit(VCSMixin, unittest.TestCase):
self.vcs._run('git', 'submodule', 'update', '--init', '--recursive')
self.assertEqual(
get_vcs_files(self.ui),
[fn.replace('/', os.path.sep) for fn in [
[
'.gitmodules',
'file5',
'sub1',
'sub1/file1',
'sub1/file2',
'subdir',
'subdir/file6',
'subdir/sub2',
'subdir/sub2/.gitmodules',
'subdir/sub2/file3',
'subdir/sub2/sub3',
'subdir/sub2/sub3/file4',
]])
])
class BzrHelper(VCSHelper):
@ -1251,9 +1204,14 @@ class SvnHelper(VCSHelper):
self._run('svn', 'co', 'file:///' + os.path.abspath('repo').replace(os.path.sep, '/'), 'checkout')
os.chdir('checkout')
def _add_directories_and_sort(self, filelist):
from check_manifest import normalize_names
names = set(normalize_names(filelist))
names.update([posixpath.dirname(fn) for fn in names])
return sorted(names - {''})
def _add_to_vcs(self, filenames):
from check_manifest import add_directories_and_sort
self._run('svn', 'add', '-N', '--', *add_directories_and_sort(filenames))
self._run('svn', 'add', '-N', '--', *self._add_directories_and_sort(filenames))
def _commit(self):
self._run('svn', 'commit', '-m', 'Initial')
@ -1271,9 +1229,8 @@ class TestSvn(VCSMixin, unittest.TestCase):
self.vcs._run('svn', 'up')
self._create_files(['a.txt', 'ext/b.txt'])
self.vcs._run('svn', 'add', 'a.txt', 'ext/b.txt')
j = os.path.join
self.assertEqual(get_vcs_files(self.ui),
['a.txt', 'ext', j('ext', 'b.txt')])
['a.txt', 'ext/b.txt'])
class TestSvnExtraErrors(unittest.TestCase):
@ -1394,15 +1351,182 @@ class TestUserInterface(unittest.TestCase):
class TestIgnoreList(unittest.TestCase):
def setUp(self):
from check_manifest import IgnoreList
self.ignore = IgnoreList()
def test_repr(self):
from check_manifest import IgnoreList
ignore = IgnoreList(['a'], ['b'])
self.assertEqual(repr(ignore), "IgnoreList(['a'], ['b'])")
ignore = IgnoreList()
self.assertEqual(repr(ignore), "IgnoreList([])")
def test_exclude_pattern(self):
self.ignore.exclude('*.txt')
self.assertEqual(self.ignore.filter([
'foo.md',
'bar.txt',
'subdir/bar.txt',
]), [
'foo.md',
'subdir/bar.txt',
])
def test_failed_addition(self):
def test_exclude_file(self):
self.ignore.exclude('bar.txt')
self.assertEqual(self.ignore.filter([
'foo.md',
'bar.txt',
'subdir/bar.txt',
]), [
'foo.md',
'subdir/bar.txt',
])
def test_exclude_doest_apply_to_directories(self):
self.ignore.exclude('subdir')
self.assertEqual(self.ignore.filter([
'foo.md',
'subdir/bar.txt',
]), [
'foo.md',
'subdir/bar.txt',
])
def test_global_exclude(self):
self.ignore.global_exclude('a*.txt')
self.assertEqual(self.ignore.filter([
'bar.txt', # make sure full filenames are matched
'afile.txt',
'subdir/afile.txt',
'adir/file.txt', # make sure * doesn't match /
]), [
# no bar.txt! because https://bugs.python.org/issue14106
'adir/file.txt',
])
def test_global_exclude_does_not_apply_to_directories(self):
self.ignore.global_exclude('subdir')
self.assertEqual(self.ignore.filter([
'bar.txt',
'subdir/afile.txt',
]), [
'bar.txt',
'subdir/afile.txt',
])
def test_recursive_exclude(self):
self.ignore.recursive_exclude('subdir', 'a*.txt')
self.assertEqual(self.ignore.filter([
'afile.txt',
'subdir/afile.txt',
'subdir/extra/afile.txt',
'subdir/adir/file.txt',
'other/afile.txt',
]), [
'afile.txt',
'subdir/adir/file.txt',
'other/afile.txt',
])
def test_recursive_exclude_does_not_apply_to_directories(self):
self.ignore.recursive_exclude('subdir', 'dir')
self.assertEqual(self.ignore.filter([
'afile.txt',
'subdir/dir/afile.txt',
]), [
'afile.txt',
'subdir/dir/afile.txt',
])
def test_recursive_exclude_can_prune(self):
self.ignore.recursive_exclude('subdir', '*')
self.assertEqual(self.ignore.filter([
'afile.txt',
'subdir/afile.txt',
'subdir/dir/afile.txt',
'subdir/dir/dir/afile.txt',
]), [
'afile.txt',
])
def test_prune(self):
self.ignore.prune('subdir')
self.assertEqual(self.ignore.filter([
'foo.md',
'subdir/bar.txt',
'unrelated/subdir/baz.txt',
]), [
'foo.md',
'unrelated/subdir/baz.txt',
])
def test_prune_subdir(self):
self.ignore.prune('a/b')
self.assertEqual(self.ignore.filter([
'foo.md',
'a/b/bar.txt',
'a/c/bar.txt',
]), [
'foo.md',
'a/c/bar.txt',
])
def test_prune_glob(self):
self.ignore.prune('su*r')
self.assertEqual(self.ignore.filter([
'foo.md',
'subdir/bar.txt',
'unrelated/subdir/baz.txt',
]), [
'foo.md',
'unrelated/subdir/baz.txt',
])
def test_prune_glob_is_not_too_greedy(self):
self.ignore.prune('su*r')
self.assertEqual(self.ignore.filter([
'foo.md',
# super-unrelated/subdir matches su*r if you allow * to match /,
# which fnmatch does!
'super-unrelated/subdir/qux.txt',
]), [
'foo.md',
'super-unrelated/subdir/qux.txt',
])
def test_default_excludes_pkg_info(self):
from check_manifest import IgnoreList
ignore = IgnoreList.default()
self.assertEqual(ignore.filter([
'PKG-INFO',
'bar.txt',
]), [
'bar.txt',
])
def test_default_excludes_egg_info(self):
from check_manifest import IgnoreList
ignore = IgnoreList.default()
self.assertEqual(ignore.filter([
'mypackage.egg-info/PKG-INFO',
'mypackage.egg-info/SOURCES.txt',
'mypackage.egg-info/requires.txt',
'bar.txt',
]), [
'bar.txt',
])
def test_default_excludes_egg_info_in_a_subdirectory(self):
from check_manifest import IgnoreList
with self.assertRaises(TypeError):
IgnoreList() + 42
ignore = IgnoreList.default()
self.assertEqual(ignore.filter([
'src/mypackage.egg-info/PKG-INFO',
'src/mypackage.egg-info/SOURCES.txt',
'src/mypackage.egg-info/requires.txt',
'bar.txt',
]), [
'bar.txt',
])
def pick_installed_vcs():
@ -1520,7 +1644,8 @@ class TestCheckManifest(unittest.TestCase):
from check_manifest import check_manifest, IgnoreList
self._create_repo_with_code()
self._add_to_vcs('unrelated.txt')
self.assertTrue(check_manifest(extra_ignore=IgnoreList(['*.txt'])),
ignore = IgnoreList().global_exclude('*.txt')
self.assertTrue(check_manifest(extra_ignore=ignore),
sys.stderr.getvalue())
def test_suggestions(self):
@ -1630,7 +1755,8 @@ class TestCheckManifest(unittest.TestCase):
self._add_to_vcs('foo.egg-info')
self._add_to_vcs('moo.mo')
self._add_to_vcs(os.path.join('subdir', 'bar.egg-info'))
self.assertFalse(check_manifest(extra_ignore_bad_ideas=IgnoreList(['*.mo'])))
ignore = IgnoreList().global_exclude('*.mo')
self.assertFalse(check_manifest(extra_ignore_bad_ideas=ignore))
self.assertIn("you have foo.egg-info in source control!",
sys.stderr.getvalue())
self.assertNotIn("moo.mo", sys.stderr.getvalue())

2
tox.ini
Unescape View File

@ -1,6 +1,6 @@
[tox]
envlist =
py27,py35,py36,py37,py38,pypy,pypy3,flake8
py35,py36,py37,py38,pypy3,flake8
[testenv]
passenv = LANG LC_CTYPE LC_ALL MSYSTEM

Write Preview
Loading…
Cancel
Save