Add syntax highlighting to blog posts

pull/162/head
Adam Johnson 2 years ago
parent 264c06fd38
commit b4dbbe397e
  1. 118
      blog/01-how-you-implemented-your-python-decorator-is-wrong.md
  2. 130
      blog/02-the-interaction-between-decorators-and-descriptors.md
  3. 114
      blog/03-implementing-a-factory-for-creating-decorators.md
  4. 222
      blog/04-implementing-a-universal-decorator.md
  5. 82
      blog/05-decorators-which-accept-arguments.md
  6. 88
      blog/06-maintaining-decorator-state-using-a-class.md
  7. 183
      blog/07-the-missing-synchronized-decorator.md
  8. 92
      blog/08-the-synchronized-decorator-as-context-manager.md
  9. 48
      blog/09-performance-overhead-of-using-decorators.md
  10. 22
      blog/10-performance-overhead-when-applying-decorators-to-methods.md
  11. 28
      blog/11-safely-applying-monkey-patches-in-python.md
  12. 36
      blog/12-using-wrapt-to-support-testing-of-software.md
  13. 37
      blog/13-ordering-issues-when-monkey-patching-in-python.md
  14. 32
      blog/14-automatic-patching-of-python-applications.md

@ -72,17 +72,17 @@ Basics of a Python decorator
Everyone should know what the Python decorator syntax is.
```
```python
@function_wrapper
def function():
pass
```
The ``@`` annotation to denote the application of a decorator was only
The `@` annotation to denote the application of a decorator was only
added in Python 2.4. It is actually though only fancy syntactic sugar. It
is actually equivalent to writing:
```
```python
def function():
pass
@ -108,7 +108,7 @@ Although I mentioned using function closures to implement a decorator, to
understand how the more generic case of a function wrapper works it is more
illustrative to show how to implement it using a class.
```
```python
class function_wrapper(object):
def __init__(self, wrapped):
self.wrapped = wrapped
@ -122,14 +122,14 @@ def function():
The class instance in this example is initialised with and records the
original function object. When the now wrapped function is called, it is
actually the ``__call__()`` method of the wrapper object which is invoked.
actually the `__call__()` method of the wrapper object which is invoked.
This in turn would then call the original wrapped function.
Simply passing through the call to the wrapper alone isn't particularly
useful, so normally you would actually want to do some work either before
or after the wrapped function is called. Or you may want to modify the
input arguments or the result as they pass through the wrapper. This is
just a matter of modifying the ``__call__()`` method appropriately to do
just a matter of modifying the `__call__()` method appropriately to do
what you want.
Using a class to implement the wrapper for a decorator isn't actually that
@ -139,11 +139,11 @@ the decorator function. When the now wrapped function is called, the nested
function is actually being called. This in turn would again then call the
original wrapped function.
```
```python
def function_wrapper(wrapped):
def _wrapper(*args, **kwargs):
return wrapped(*args, **kwargs)
return _wrapper
return _wrapper
@function_wrapper
def function():
@ -161,39 +161,43 @@ Introspecting a function
Now when we talk about functions, we expect them to specify properties
which describe them as well as document what they do. These include the
``__name__`` and ``__doc__`` attributes. When we use a wrapper though, this
`__name__` and `__doc__` attributes. When we use a wrapper though, this
no longer works as we expect as in the case of using a function closure,
the details of the nested function are returned.
```
```python
def function_wrapper(wrapped):
def _wrapper(*args, **kwargs):
return wrapped(*args, **kwargs)
return _wrapper
return _wrapper
@function_wrapper
def function():
pass
pass
```
```pycon
>>> print(function.__name__)
_wrapper
```
If we use a class to implement the wrapper, as class instances do not
normally have a ``__name__`` attribute, attempting to access the name of
normally have a `__name__` attribute, attempting to access the name of
the function will actually result in an AttributeError exception.
```
```python
class function_wrapper(object):
def __init__(self, wrapped):
self.wrapped = wrapped
def __call__(self, *args, **kwargs):
return self.wrapped(*args, **kwargs)
return self.wrapped(*args, **kwargs)
@function_wrapper
def function():
pass
pass
```
```pycon
>>> print(function.__name__)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
@ -205,17 +209,17 @@ of interest from the wrapped function to the nested wrapper function. This
will then result in the function name and documentation strings being
correct.
```
```python
def function_wrapper(wrapped):
def _wrapper(*args, **kwargs):
return wrapped(*args, **kwargs)
_wrapper.__name__ = wrapped.__name__
_wrapper.__doc__ = wrapped.__doc__
return _wrapper
return _wrapper
@function_wrapper
def function():
pass
pass
>>> print(function.__name__)
function
@ -223,34 +227,36 @@ function
Needing to manually copy the attributes is laborious, and would need to be
updated if any further special attributes were added which needed to be
copied. For example, we should also copy the ``__module__`` attribute, and
in Python 3 the ``__qualname__`` and ``__annotations__`` attributes were
copied. For example, we should also copy the `__module__` attribute, and
in Python 3 the `__qualname__` and `__annotations__` attributes were
added. To aid in getting this right, the Python standard library provides
the ``functools.wraps()`` decorator which does this task for you.
the `functools.wraps()` decorator which does this task for you.s
```
import functools
```python
import functools
def function_wrapper(wrapped):
@functools.wraps(wrapped)
def _wrapper(*args, **kwargs):
return wrapped(*args, **kwargs)
return _wrapper
return _wrapper
@function_wrapper
def function():
pass
pass
```
```pycon
>>> print(function.__name__)
function
```
If using a class to implement the wrapper, instead of the
``functools.wraps()`` decorator, we would use the
``functools.update_wrapper()`` function.
`functools.wraps()` decorator, we would use the
`functools.update_wrapper()` function.
```
import functools
```python
import functools
class function_wrapper(object):
def __init__(self, wrapped):
@ -261,7 +267,7 @@ class function_wrapper(object):
```
So we might have a solution to ensuring the function name and any
documentation string is correct in the form of ``functools.wraps()``, but
documentation string is correct in the form of `functools.wraps()`, but
actually we don't and this will not always work as I will show below.
Now what if we want to query the argument specification for a function.
@ -270,14 +276,16 @@ wrapped function, it returns that of the wrapper. In the case of using a
function closure, this is the nested function. The decorator is therefore
not signature preserving.
```
import inspect
```python
import inspect
def function_wrapper(wrapped): ...
@function_wrapper
def function(arg1, arg2): pass
def function(arg1, arg2): pass
```
```pycon
>>> print(inspect.getargspec(function))
ArgSpec(args=[], varargs='args', keywords='kwargs', defaults=None)
```
@ -287,12 +295,14 @@ get an exception complaining that the wrapped function isn't actually a
function. As a result it isn't possible to derive an argument specification
at all, even though the wrapped function is actually still callable.
```
class function_wrapper(object): ...
```python
class function_wrapper(object): ...
@function_wrapper
def function(arg1, arg2): pass
def function(arg1, arg2): pass
```
```pycon
>>> print(inspect.getargspec(function))
Traceback (most recent call last):
File "...", line XXX, in <module>
@ -303,7 +313,7 @@ TypeError: <__main__.function_wrapper object at 0x107e0ac90> is not a Python fun
```
Another example of introspection one can do is to use
``inspect.getsource()`` to get back the source code related to a function.
`inspect.getsource()` to get back the source code related to a function.
This also will fail, with it giving the source code for the nested wrapper
function in the case of a function closure and again failing outright with
an exception in the case of the class based wrapper.
@ -313,39 +323,41 @@ Wrapping class methods
Now, as well as normal functions, decorators can also be applied to methods
of classes. Python even includes a couple of special decorators called
``@classmethod`` and ``@staticmethod`` for converting normal instance
`@classmethod` and `@staticmethod` for converting normal instance
methods into these special method types. Methods of classes do provide a
number of potential problems though.
```
class Class(object):
```python
class Class(object):
@function_wrapper
def method(self):
pass
pass
@classmethod
def cmethod(cls):
pass
pass
@staticmethod
def smethod():
pass
```
The first is that even if using ``functools.wraps()`` or
``functools.update_wrapper()`` in your decorator, when the decorator is
applied around ``@classmethod`` or ``@staticmethod``, it can fail with an
The first is that even if using `functools.wraps()` or
`functools.update_wrapper()` in your decorator, when the decorator is
applied around ``@classmethod` or `@staticmethod`, it can fail with an
exception. This is because the wrappers created by these, do not have some
of the attributes being copied.
```
```python
class Class(object):
@function_wrapper
@classmethod
def cmethod(cls):
pass
pass
```
```
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in Class
@ -364,14 +376,14 @@ callable. This need not actually be the case. A wrapped function can
actually be what is called a descriptor, meaning that in order to get back
a callable, the descriptor has to be correctly bound to the instance first.
```
```python
class Class(object):
@function_wrapper
@classmethod
def cmethod(cls):
pass
pass
>>> Class.cmethod()
>>> Class.cmethod()
Traceback (most recent call last):
File "classmethod.py", line 15, in <module>
Class.cmethod()
@ -388,12 +400,12 @@ doesn't mean they are necessarily correct and will always work.
The issues highlighted so far are:
* Preservation of function ``__name__`` and ``__doc__``.
* Preservation of function `__name__` and `__doc__`.
* Preservation of function argument specification.
* Preservation of ability to get function source code.
* Ability to apply decorators on top of other decorators that are implemented as descriptors.
The ``functools.wraps()`` function is given as a solution to the first but
The `functools.wraps()` function is given as a solution to the first but
doesn't always work, at least in Python 2. It doesn't help at all though
with preserving the introspection of a functions argument specification and
ability to get the source code for a function.

@ -9,13 +9,13 @@ the first post titled [How you implemented your Python decorator is wrong](
In that first post I described a number of ways in which the traditional
way that Python decorators are implemented is lacking. These were:
* Preservation of function ``__name__`` and ``__doc__``.
* Preservation of function `__name__` and `__doc__`.
* Preservation of function argument specification.
* Preservation of ability to get function source code.
* Ability to apply decorators on top of other decorators that are implemented as descriptors.
I described previously how ``functools.wraps()`` attempts to solve the problem
with preservation of the introspection of the ``__name__`` and ``__doc__``
I described previously how `functools.wraps()` attempts to solve the problem
with preservation of the introspection of the `__name__` and `__doc__`
attributes, but highlight one case in Python 2 where it can fail, and also
note that it doesn't help with the preservation of the function argument
specification nor the ability to access the source code.
@ -33,16 +33,16 @@ reading up about them elsewhere.
In short though, a descriptor is an object attribute with binding
behaviour, one whose attribute access has been overridden by methods in the
descriptor protocol. Those methods are ``__get__()``, ``__set__()``, and
``__delete__()``. If any of those methods are defined for an object, it is
descriptor protocol. Those methods are `__get__()`, `__set__()`, and
`__delete__()`. If any of those methods are defined for an object, it is
said to be a descriptor.
* ``obj.attribute``
--> ``attribute.__get__(obj, type(obj))``
* ``obj.attribute = value``
--> ``attribute.__set__(obj, value)``
* ``del obj.attribute``
--> ``attribute.__delete__(obj)``
* `obj.attribute`
--> `attribute.__get__(obj, type(obj))`
* `obj.attribute = value`
--> `attribute.__set__(obj, value)`
* `del obj.attribute`
--> `attribute.__delete__(obj)`
What this means is that if an attribute of a class has any of these special
methods defined, when the corresponding operation is performed on that
@ -54,17 +54,17 @@ You may well be thinking that you have never made use of descriptors, but
fact is that function objects are actually descriptors. When a function is
originally added to a class definition it is as a normal function. When you
access that function using a dotted attribute path, you are invoking the
``__get__()`` method to bind the function to the class instance, turning it
`__get__()` method to bind the function to the class instance, turning it
into a bound method of that object.
```
def f(obj): pass
```pycon
>>> def f(obj): pass
>>> hasattr(f, '__get__')
True
True
>>> f
<function f at 0x10e963cf8>
<function f at 0x10e963cf8>
>>> obj = object()
@ -72,17 +72,17 @@ True
<bound method object.f of <object object at 0x10e8ac0b0>>
```
So when calling a method of a class, it is not the ``__call__()`` method of
the original function object that is called, but the ``__call__()`` method
So when calling a method of a class, it is not the `__call__()` method of
the original function object that is called, but the `__call__()` method
of the temporary bound object that is created as a result of accessing the
function.
You of course don't usually see all these intermediary steps and just see
the outcome.
```
```pycon
>>> class Object(object):
... def f(self): pass
... def f(self): pass
>>> obj = Object()
@ -93,14 +93,16 @@ the outcome.
Looking back now at the example given in the first blog post where we
wrapped a decorator around a class method, we encountered the error:
```
```python
class Class(object):
@function_wrapper
@classmethod
def cmethod(cls):
pass
pass
```
>>> Class.cmethod()
```pycon
>>> Class.cmethod()
Traceback (most recent call last):
File "classmethod.py", line 15, in <module>
Class.cmethod()
@ -109,21 +111,21 @@ Traceback (most recent call last):
TypeError: 'classmethod' object is not callable
```
The problem with this example was that for the ``@classmethod`` decorator
The problem with this example was that for the `@classmethod` decorator
to work correctly, it is dependent on the descriptor protocol being applied
properly. This is because the ``__call__()`` method only exists on the
result returned by ``__get__()`` when it is called, there is no
``__call__()`` method on the ``@classmethod`` decorator itself.
properly. This is because the `__call__()` method only exists on the
result returned by `__get__()` when it is called, there is no
`__call__()` method on the `@classmethod` decorator itself.
More specifically, the simple type of decorator that people normally use is
not itself honouring the descriptor protocol and applying that to the
wrapped object to yield the bound function object which should actually be
called. Instead it is simply calling the wrapped object directly, which
will fail if it doesn't have a ``__call__()``.
will fail if it doesn't have a `__call__()`.
Why then does applying a decorator to a normal instance method still work?
This still works because a normal function still has a ``__call__()``
This still works because a normal function still has a `__call__()`
method. In bypassing the descriptor protocol of the wrapped function it is
calling this. Although the binding protocol is side stepped, things still
work out because the wrapper will pass the 'self' argument for the instance
@ -132,7 +134,7 @@ object.
For a normal instance method the result in this situation is effectively
the same. It only falls apart when the wrapped object, as in the case of
``@classmethod``, are dependent on the descriptor protocol being applied
`@classmethod`, are dependent on the descriptor protocol being applied
correctly.
Wrappers as descriptors
@ -142,27 +144,27 @@ The way to solve this problem where the wrapper is not honouring the
descriptor protocol and performing binding on the wrapped object in the
case of a method on a class, is for wrappers to also be descriptors.
```
class bound_function_wrapper(object):
```python
class bound_function_wrapper(object):
def __init__(self, wrapped):
self.wrapped = wrapped
self.wrapped = wrapped
def __call__(self, *args, **kwargs):
return self.wrapped(*args, **kwargs)
return self.wrapped(*args, **kwargs)
class function_wrapper(object):
class function_wrapper(object):
def __init__(self, wrapped):
self.wrapped = wrapped
self.wrapped = wrapped
def __get__(self, instance, owner):
wrapped = self.wrapped.__get__(instance, owner)
return bound_function_wrapper(wrapped)
return bound_function_wrapper(wrapped)
def __call__(self, *args, **kwargs):
return self.wrapped(*args, **kwargs)
```
If the wrapper is applied to a normal function, the ``__call__()`` method
If the wrapper is applied to a normal function, the `__call__()` method
of the wrapper is used. If the wrapper is applied to a method of a class,
the ``__get__()`` method is called, which returns a new bound wrapper and
the ``__call__()`` method of that is invoked instead. This allows our
the `__get__()` method is called, which returns a new bound wrapper and
the `__call__()` method of that is invoked instead. This allows our
wrapper to be used around descriptors as it propagates the descriptor
protocol.
@ -174,29 +176,29 @@ protocol as shown.
The question now is how do we address the other issues that were listed.
We solved naming using ``functools.wrap()``/``functools.update_wrapper()``
We solved naming using `functools.wrap()`/`functools.update_wrapper()`
before, but what do they do and can we still use them.
Well ``wraps()`` just uses ``update_wrapper()``, so we just need to look at
Well `wraps()` just uses `update_wrapper()`, so we just need to look at
it.
```
```python
WRAPPER_ASSIGNMENTS = ('__module__',
'__name__', '__qualname__', '__doc__',
'__annotations__')
WRAPPER_UPDATES = ('__dict__',)
WRAPPER_UPDATES = ('__dict__',)
def update_wrapper(wrapper, wrapped,
assigned = WRAPPER_ASSIGNMENTS,
updated = WRAPPER_UPDATES):
wrapper.__wrapped__ = wrapped
updated = WRAPPER_UPDATES):
wrapper.__wrapped__ = wrapped
for attr in assigned:
try:
value = getattr(wrapped, attr)
except AttributeError:
pass
else:
setattr(wrapper, attr, value)
setattr(wrapper, attr, value)
for attr in updated:
getattr(wrapper, attr).update(
getattr(wrapped, attr, {}))
@ -206,12 +208,12 @@ What is shown here is what is in Python 3.3, although that actually has a
bug in it, which is fixed in Python 3.4. :-)
Looking at the body of the function, three things are being done. First off
a reference to the wrapped function is saved as ``__wrapped__``. This is the
a reference to the wrapped function is saved as `__wrapped__`. This is the
bug, as it should be done last.
The second is to copy those attributes such as ``__name__`` and ``__doc__``.
The second is to copy those attributes such as `__name__` and `__doc__`.
Finally the third thing is to copy the contents of ``__dict__`` from the
Finally the third thing is to copy the contents of `__dict__` from the
wrapped function into the wrapper, which could actually result in quite a
lot of objects needing to be copied.
@ -221,11 +223,11 @@ is able to be done at the point that the decorator is applied.
With the wrapper being a descriptor though, it technically now also needs
to be done in the bound wrapper.
```
```python
class bound_function_wrapper(object):
def __init__(self, wrapped):
self.wrapped = wrapped
functools.update_wrapper(self, wrapped)
functools.update_wrapper(self, wrapped)
class function_wrapper(object):
def __init__(self, wrapped):
@ -244,19 +246,19 @@ The solution to the performance issue is to use what is called an object
proxy. This is a special wrapper class which looks and behaves like what it
wraps.
```
class object_proxy(object):
```python
class object_proxy(object):
def __init__(self, wrapped):
self.wrapped = wrapped
try:
self.__name__= wrapped.__name__
except AttributeError:
pass
pass
@property
def __class__(self):
return self.wrapped.__class__
return self.wrapped.__class__
def __getattr__(self, name):
return getattr(self.wrapped, name)
@ -273,21 +275,21 @@ of monkey patching.
In short though, it copies limited attributes from the wrapped object to
itself, and otherwise uses special methods, properties and
``__getattr__()`` to fetch attributes from the wrapped object only when
`__getattr__()` to fetch attributes from the wrapped object only when
required thereby avoiding the need to copy across lots of attributes which
may never actually be accessed.
What we now do is derive our wrapper class from the object proxy and do
away with calling ``update_wrapper()``.
away with calling `update_wrapper()`.
```
```python
class bound_function_wrapper(object_proxy):
def __init__(self, wrapped):
super(bound_function_wrapper, self).__init__(wrapped)
def __call__(self, *args, **kwargs):
return self.wrapped(*args, **kwargs)
return self.wrapped(*args, **kwargs)
class function_wrapper(object_proxy):
@ -296,19 +298,19 @@ class function_wrapper(object_proxy):
def __get__(self, instance, owner):
wrapped = self.wrapped.__get__(instance, owner)
return bound_function_wrapper(wrapped)
return bound_function_wrapper(wrapped)
def __call__(self, *args, **kwargs):
return self.wrapped(*args, **kwargs)
return self.wrapped(*args, **kwargs)
```
In doing this, attributes like ``__name__`` and ``__doc__``, when queried
In doing this, attributes like `__name__` and `__doc__`, when queried
from the wrapper, return the values from the wrapped function. We don't
therefore as a result have the problem we did before where details were
being returned from the wrapper instead.
Using a transparent object proxy in this way also means that calls like
``inspect.getargspec()`` and ``inspect.getsource()`` will now work and
`inspect.getargspec()` and `inspect.getsource()` will now work and
return what we expect. So we have actually managed to solve those two
problems at the same time without any extra effort, which is a bonus.

@ -13,7 +13,7 @@ In the very first post I described a number of ways in which the
traditional way that Python decorators are implemented is lacking. These
were:
* Preservation of function ``__name__`` and ``__doc__``.
* Preservation of function `__name__` and `__doc__`.
* Preservation of function argument specification.
* Preservation of ability to get function source code.
* Ability to apply decorators on top of other decorators that are implemented as descriptors.
@ -44,19 +44,19 @@ Pattern for implementing the decorator
Just to refresh where we got to last time, we had an implementation of an
object proxy as:
```
class object_proxy(object):
```python
class object_proxy(object):
def __init__(self, wrapped):
self.wrapped = wrapped
try:
self.__name__= wrapped.__name__
except AttributeError:
pass
pass
@property
def __class__(self):
return self.wrapped.__class__
return self.wrapped.__class__
def __getattr__(self, name):
return getattr(self.wrapped, name)
@ -69,32 +69,32 @@ of monkey patching.
The decorator itself would then be implemented per the pattern:
```
class bound_function_wrapper(object_proxy):
```python
class bound_function_wrapper(object_proxy):
def __init__(self, wrapped):
super(bound_function_wrapper, self).__init__(wrapped)
super(bound_function_wrapper, self).__init__(wrapped)
def __call__(self, *args, **kwargs):
return self.wrapped(*args, **kwargs)
return self.wrapped(*args, **kwargs)
class function_wrapper(object_proxy):
class function_wrapper(object_proxy):
def __init__(self, wrapped):
super(function_wrapper, self).__init__(wrapped)
super(function_wrapper, self).__init__(wrapped)
def __get__(self, instance, owner):
wrapped = self.wrapped.__get__(instance, owner)
return bound_function_wrapper(wrapped)
return bound_function_wrapper(wrapped)
def __call__(self, *args, **kwargs):
return self.wrapped(*args, **kwargs)
return self.wrapped(*args, **kwargs)
```
When the wrapper is applied to a normal function, the ``__call__()`` method
When the wrapper is applied to a normal function, the `__call__()` method
of the wrapper is used. If the wrapper is applied to a method of a class,
the ``__get__()`` method is called when the attribute is accessed, which
returns a new bound wrapper and the ``__call__()`` method of that is
the `__get__()` method is called when the attribute is accessed, which
returns a new bound wrapper and the `__call__()` method of that is
invoked instead when a call is made. This allows our wrapper to be used
around descriptors as it propagates the descriptor protocol, also binding
the wrapped object as necessary.
@ -109,10 +109,10 @@ decorator to help us create decorators. This would reduce the code we need
to write for each decorator to a single function, allowing us to simplify
the code to just:
```
```python
@decorator
def my_function_wrapper(wrapped, args, kwargs):
return wrapped(*args, **kwargs)
return wrapped(*args, **kwargs)
@my_function_wrapper
def function():
@ -122,11 +122,11 @@ def function():
What would this decorator factory need to look like?
As it turns out, our decorator factory is quite simple and isn't really
much different to using a ``partial()``, combining our new wrapper argument
much different to using a `partial()`, combining our new wrapper argument
from when the decorator is defined, with the wrapped function when the
decorator is used and passing them into our function wrapper object.
```
```python
def decorator(wrapper):
@functools.wraps(wrapper)
def _decorator(wrapped):
@ -138,31 +138,31 @@ We now just need to amend our function wrapper implementation to delegate
the actual execution of the wrapped object to the user supplied decorator
wrapper function.
```
class bound_function_wrapper(object_proxy):
```python
class bound_function_wrapper(object_proxy):
def __init__(self, wrapped, wrapper):
super(bound_function_wrapper, self).__init__(wrapped)
self.wrapper = wrapper
self.wrapper = wrapper
def __call__(self, *args, **kwargs):
return self.wrapper(self.wrapped, args, kwargs)
return self.wrapper(self.wrapped, args, kwargs)
class function_wrapper(object_proxy):
class function_wrapper(object_proxy):
def __init__(self, wrapped, wrapper):
super(function_wrapper, self).__init__(wrapped)
self.wrapper = wrapper
self.wrapper = wrapper
def __get__(self, instance, owner):
wrapped = self.wrapped.__get__(instance, owner)
return bound_function_wrapper(wrapped, self.wrapper)
return bound_function_wrapper(wrapped, self.wrapper)
def __call__(self, *args, **kwargs):
return self.wrapper(self.wrapped, args, kwargs)
```
The ``__call__()`` method of our function wrapper, for when it is used
The `__call__()` method of our function wrapper, for when it is used
around a normal function, now just calls the user supplied decorator
wrapper function with the wrapped function and arguments, leaving the
calling of the wrapped function up to the user supplied decorator wrapper
@ -170,7 +170,7 @@ function.
In the case where binding a function, the wrapper is also passed to the
bound wrapper. The bound wrapper is more or less the same, with the
``__call__()`` method delegating to the user supplied decorator wrapper
`__call__()` method delegating to the user supplied decorator wrapper
function.
So we can make creating decorators easier using a factory. Lets now check
@ -188,7 +188,7 @@ To test out how our new decorator works, we can print out the args passed
to the wrapper when the wrapped function is called and can compare the
results.
```
```python
@decorator
def my_function_wrapper(wrapped, args, kwargs):
print('ARGS', args)
@ -197,11 +197,13 @@ def my_function_wrapper(wrapped, args, kwargs):
First up lets try wrapping a normal function:
```
```python
@my_function_wrapper
def function(a, b):
pass
pass
```
```pycon
>>> function(1, 2)
ARGS (1, 2)
```
@ -211,13 +213,13 @@ called are output.
What about when wrapping an instance method?
```
```python
class Class(object):
@my_function_wrapper
def function_im(self, a, b):
pass
pass
c = Class()
c = Class()
>>> c.function_im()
ARGS (1, 2)
@ -234,29 +236,29 @@ class as it is now associated with the bound function passed in, rather
than the argument list.
To solve this problem we can remember what the instance was that was passed
to the ``__get__()`` method when it was called to bind the function. This
to the `__get__()` method when it was called to bind the function. This
can then be passed through to the bound wrapper when it is created.
```
class bound_function_wrapper(object_proxy):
```python
class bound_function_wrapper(object_proxy):
def __init__(self, wrapped, instance, wrapper):
super(bound_function_wrapper, self).__init__(wrapped)
self.instance = instance
self.wrapper = wrapper
self.wrapper = wrapper
def __call__(self, *args, **kwargs):
return self.wrapper(self.wrapped, self.instance, args, kwargs)
return self.wrapper(self.wrapped, self.instance, args, kwargs)
class function_wrapper(object_proxy):
class function_wrapper(object_proxy):
def __init__(self, wrapped, wrapper):
super(function_wrapper, self).__init__(wrapped)
self.wrapper = wrapper
self.wrapper = wrapper
def __get__(self, instance, owner):
wrapped = self.wrapped.__get__( instance, owner)
return bound_function_wrapper(wrapped, instance, self.wrapper)
return bound_function_wrapper(wrapped, instance, self.wrapper)
def __call__(self, *args, **kwargs):
return self.wrapper(self.wrapped, None, args, kwargs)
@ -270,16 +272,18 @@ new instance argument.
We can now modify our wrapper function for the decorator to output both the
instance and the arguments passed.
```
```python
@decorator
def my_function_wrapper(wrapped, instance, args, kwargs):
print('INSTANCE', instance)
print('ARGS', args)
return wrapped(*args, **kwargs)
return wrapped(*args, **kwargs)
```
```pycon
>>> function(1, 2)
INSTANCE None
ARGS (1, 2)
ARGS (1, 2)
>>> c.function_im(1, 2)
INSTANCE <__main__.Class object at 0x1085ca9d0>
@ -297,7 +301,7 @@ which we still need to check. This is calling an instance method by calling
the function on the class and passing the object instance explicitly as the
first argument.
```
```pycon
>>> Class.function_im(c, 1, 2)
INSTANCE None
ARGS (<__main__.Class object at 0x1085ca9d0>, 1, 2)
@ -314,21 +318,21 @@ calling the decorator wrapper function and pop the instance off the start
of the argument list. We then use a partial to bind the instance to the
wrapped function ourselves and call the decorator wrapper function.
```
class bound_function_wrapper(object_proxy):
```python
class bound_function_wrapper(object_proxy):
def __call__(self, *args, **kwargs):
if self.instance is None:
instance, args = args[0], args[1:]
wrapped = functools.partial(self.wrapped, instance)
return self.wrapper(wrapped, instance, args, kwargs)
return self.wrapper(wrapped, instance, args, kwargs)
return self.wrapper(self.wrapped, self.instance, args, kwargs)
```
We then get the same result no matter whether the instance method is called
via the class or not.
```
```pycon
>>> Class.function_im(c, 1, 2)
INSTANCE <__main__.Class object at 0x1085ca9d0>
ARGS (1, 2)
@ -343,14 +347,16 @@ decorator was applied to an instance method of a class.
What about other method types that a class can have, specifically class
method and static methods.
```
```python
class Class(object):
@my_function_wrapper
@classmethod
def function_cm(cls, a, b):
pass
pass
```
```pycon
>>> Class.function_cm(1, 2)
INSTANCE 1
ARGS (2,)
@ -385,7 +391,7 @@ be used in a different context.
What I am instead aiming for is the ability to do:
```
```python
@decorator
def universal(wrapped, instance, args, kwargs):
if instance is None:

@ -43,30 +43,30 @@ Normal functions vs instance methods
The pattern for our universal decorator as described so far was as follows:
```
class bound_function_wrapper(object_proxy):
```python
class bound_function_wrapper(object_proxy):
def __init__(self, wrapped, instance, wrapper):
super(bound_function_wrapper, self).__init__(wrapped)
self.instance = instance
self.wrapper = wrapper
self.wrapper = wrapper
def __call__(self, *args, **kwargs):
if self.instance is None:
instance, args = args[0], args[1:]
wrapped = functools.partial(self.wrapped, instance)
return self.wrapper(wrapped, instance, args, kwargs)
return self.wrapper(self.wrapped, self.instance, args, kwargs)
return self.wrapper(self.wrapped, self.instance, args, kwargs)
class function_wrapper(object_proxy):
class function_wrapper(object_proxy):
def __init__(self, wrapped, wrapper):
super(function_wrapper, self).__init__(wrapped)
self.wrapper = wrapper
self.wrapper = wrapper
def __get__(self, instance, owner):
wrapped = self.wrapped.__get__(instance, owner)
return bound_function_wrapper(wrapped, instance, self.wrapper)
return bound_function_wrapper(wrapped, instance, self.wrapper)
def __call__(self, *args, **kwargs):
return self.wrapper(self.wrapped, None, args, kwargs)
@ -74,7 +74,7 @@ class function_wrapper(object_proxy):
This was used in conjunction with our decorator factory:
```
```python
def decorator(wrapper):
@functools.wraps(wrapper)
def _decorator(wrapped):
@ -87,90 +87,98 @@ factory to create a decorator which would dump out the values of any
instance the wrapped function is bound to, and the arguments passed to the
call when executed.
```
```python
@decorator
def my_function_wrapper(wrapped, instance, args, kwargs):
print('INSTANCE', instance)
print('ARGS', args)
return wrapped(*args, **kwargs)
return wrapped(*args, **kwargs)
```
This gave us the desired results for when the decorator was applied to a
normal function and instance method, including when an instance method was
called via the class and the instance passed in explicitly.
```
```python
@my_function_wrapper
def function(a, b):
pass
```
```pycon
>>> function(1, 2)
INSTANCE None
ARGS (1, 2)
ARGS (1, 2)
```
```python
class Class(object):
@my_function_wrapper
def function_im(self, a, b):
pass
pass
```
c = Class()
```pycon
>>> c = Class()
>>> c.function_im(1, 2)
INSTANCE <__main__.Class object at 0x1085ca9d0>
ARGS (1, 2)
ARGS (1, 2)
>>> Class.function_im(c, 1, 2)
INSTANCE <__main__.Class object at 0x1085ca9d0>
ARGS (1, 2)
ARGS (1, 2)
```
The change to support the latter however, broke things for the case of the
decorator being applied to a class method. Similarly for a static method.
```
```python
class Class(object):
@my_function_wrapper
@classmethod
def function_cm(self, a, b):
pass
pass
@my_function_wrapper
@staticmethod
def function_sm(a, b):
pass
```
```pycon
>>> Class.function_cm(1, 2)
INSTANCE 1
ARGS (2,)
ARGS (2,)
>>> Class.function_sm(1, 2)
INSTANCE 1
ARGS (2,)
ARGS (2,)
```
Class methods and static methods
--------------------------------
The point we are at therefore, is that in the case where the instance is
passed as ``None``, we need to be able to distinguish between the three
passed as `None`, we need to be able to distinguish between the three
cases of:
* an instance method being called via the class
* a class method being called
* a static method being called
One way this can be done is by looking at the ``__self__`` attribute of the
One way this can be done is by looking at the `__self__` attribute of the
bound function. This attribute will provide information about the type of
object which the function was bound to at that specific point in time. Lets
first check this out for where a method is called via the class.
```
```pycon
>>> print(Class.function_im.__self__)
None
None
>>> print(Class.function_cm.__self__)
<class '__main__.Class'>
<class '__main__.Class'>
>>> print(Class.function_sm.__self__)
Traceback (most recent call last):
@ -180,25 +188,25 @@ Traceback (most recent call last):
AttributeError: 'function' object has no attribute '__self__'
```
So for the case of calling an instance method via the class, ``__self__``
will be ``None``, for a class method it will be the class type and in the
case of a static method, there will not even be a ``__self__`` attribute.
So for the case of calling an instance method via the class, `__self__`
will be `None`, for a class method it will be the class type and in the
case of a static method, there will not even be a `__self__` attribute.
This would therefore appear to give us a way of detecting the different
cases.
Before we code up a solution based on this though, lets check with Python 3
just to be sure we are okay there and that nothing has changed.
```
```pycon
>>> print(Class.function_im.__self__)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "dectest.py", line 19, in __getattr__
return getattr(self.wrapped, name)
AttributeError: 'function' object has no attribute '__self__'
AttributeError: 'function' object has no attribute '__self__'
>>> print(Class.function_cm.__self__)
<class '__main__.Class'>
<class '__main__.Class'>
>>> print(Class.function_sm.__self__)
Traceback (most recent call last):
@ -214,8 +222,8 @@ aren't going to be able to use this approach. Why is this case?
The reason for this is that in Python 3 they decided to eliminate the idea
of an unbound method and this check was relying on the fact that when
accessing an instance method via the class, it would actually return an
instance of an unbound method for which the ``__self__`` attribute was
``None``. So although we can distinguish the case for a class method still,
instance of an unbound method for which the `__self__` attribute was
`None`. So although we can distinguish the case for a class method still,
we can now no longer distinguish the case of calling an instance method via
the class, from the case of calling a static method.
@ -228,38 +236,38 @@ the type of the wrapped object and determine if it is an instance of a
class method or static method. This information can then be passed through
to the bound function wrapper and checked.
```
class bound_function_wrapper(object_proxy):
```python
class bound_function_wrapper(object_proxy):
def __init__(self, wrapped, instance, wrapper, binding):
super(bound_function_wrapper, self).__init__(wrapped)
self.instance = instance
self.wrapper = wrapper
self.binding = binding
self.binding = binding
def __call__(self, *args, **kwargs):
if self.binding == 'function' and self.instance is None:
instance, args = args[0], args[1:]
wrapped = functools.partial(self.wrapped, instance)
return self.wrapper(wrapped, instance, args, kwargs)
return self.wrapper(self.wrapped, self.instance, args, kwargs)
return self.wrapper(wrapped, instance, args, kwargs)
return self.wrapper(self.wrapped, self.instance, args, kwargs)
class function_wrapper(object_proxy):
class function_wrapper(object_proxy):
def __init__(self, wrapped, wrapper):
super(function_wrapper, self).__init__(wrapped)
self.wrapper = wrapper
self.wrapper = wrapper
if isinstance(wrapped, classmethod):
self.binding = 'classmethod'
elif isinstance(wrapped, staticmethod):
self.binding = 'staticmethod'
else:
self.binding = 'function'
self.binding = 'function'
def __get__(self, instance, owner):
wrapped = self.wrapped.__get__(instance, owner)
return bound_function_wrapper(wrapped, instance, self.wrapper,
self.binding)
self.binding)
def __call__(self, *args, **kwargs):
return self.wrapper(self.wrapped, None, args, kwargs)
@ -274,18 +282,18 @@ decorators around them.
If someone is actually implementing the descriptor protocol in their
decorator, hopefully they would also be using an object proxy as is done
here. Because the object proxy implements ``__class__`` as a property, it
here. Because the object proxy implements `__class__` as a property, it
would return the class of the wrapped object, this should mean that an
``isinstance()`` check will still be successful as ``isinstance()`` gives
priority to what ``__class__`` yields rather than the actual type of the
`isinstance()` check will still be successful as `isinstance()` gives
priority to what `__class__` yields rather than the actual type of the
object.
Anyway, trying out our tests again with this change we get:
```
```pycon
>>> c.function_im(1,2)
INSTANCE <__main__.Class object at 0x101f973d0>
ARGS (1, 2)
ARGS (1, 2)
>>> Class.function_im(c, 1, 2)
INSTANCE <__main__.Class object at 0x101f973d0>
@ -297,11 +305,11 @@ ARGS (1, 2)
>>> Class.function_cm(1, 2)
INSTANCE None
ARGS (1, 2)
ARGS (1, 2)
>>> c.function_sm(1,2)
INSTANCE <__main__.Class object at 0x101f973d0>
ARGS (1, 2)
ARGS (1, 2)
>>> Class.function_sm(1, 2)
INSTANCE None
@ -319,30 +327,30 @@ cases.
Ideally what we want in this circumstance is that for a class method call
we want the instance argument to always be the class type, and for the case
of a static method call, for it to always be ``None``.
of a static method call, for it to always be `None`.
For the case of a static method, we could just check for 'staticmethod'
from when we checked the type of object which was wrapped.
For the case of a class method, if we look back at our test to see if we
could use the ``__self__`` attribute, what we found was that for the class
method, ``__self__`` was the class instance and for a static method the
could use the `__self__` attribute, what we found was that for the class
method, `__self__` was the class instance and for a static method the
attribute didn't exist.
What we can therefore do, is if the type of the wrapped object wasn't a
function, then we can lookup up the value of ``__self__``, defaulting to
``None`` if it doesn't exist. This one check will cater for both cases.
function, then we can lookup up the value of `__self__`, defaulting to
`None` if it doesn't exist. This one check will cater for both cases.
What we now therefore have is:
```
```python
class bound_function_wrapper(object_proxy):
def __init__(self, wrapped, instance, wrapper, binding):
super(bound_function_wrapper, self).__init__(wrapped)
self.instance = instance
self.wrapper = wrapper
self.binding = binding
self.binding = binding
def __call__(self, *args, **kwargs):
if self.binding == 'function':
@ -360,26 +368,26 @@ class bound_function_wrapper(object_proxy):
and if we run our tests one more time, we finally get the result we have
been looking for:
```
```pycon
>>> c.function_im(1,2)
INSTANCE <__main__.Class object at 0x10c2c43d0>
ARGS (1, 2)
ARGS (1, 2)
>>> Class.function_im(c, 1, 2)
INSTANCE <__main__.Class object at 0x10c2c43d0>
ARGS (1, 2)
ARGS (1, 2)
>>> c.function_cm(1,2)
INSTANCE <class '__main__.Class'>
ARGS (1, 2)
ARGS (1, 2)
>>> Class.function_cm(1, 2)
INSTANCE <class '__main__.Class'>
ARGS (1, 2)
ARGS (1, 2)
>>> c.function_sm(1,2)
INSTANCE None
ARGS (1, 2)
ARGS (1, 2)
>>> Class.function_sm(1, 2)
INSTANCE None
@ -400,7 +408,7 @@ an attribute of a class, or even an instance of a class, and then call it
via the alias so created. I only encountered this one due to some bizarre
stuff a meta class was doing.
```
```pycon
>>> Class.function_rm = Class.function_im
>>> c.function_rm(1, 2)
@ -412,13 +420,13 @@ Traceback (most recent call last):
return self.wrapper(wrapped, instance, args, kwargs)
File "test.py", line 58, in my_function_wrapper
return wrapped(*args, **kwargs)
TypeError: unbound method function_im() must be called with Class instance as first argument (got int instance instead)
TypeError: unbound method function_im() must be called with Class instance as first argument (got int instance instead)
>>> Class.function_rm = Class.function_cm
>>> c.function_rm(1, 2)
INSTANCE <class '__main__.Class'>
ARGS (1, 2)
ARGS (1, 2)
>>> Class.function_rm = Class.function_sm
>>> c.function_rm(1, 2)
@ -436,12 +444,12 @@ assigned back as an attribute of the class.
When a subsequent lookup is made via the new name, under normal
circumstances binding would occur once more to bind it to the actual
instance. In our implementation of the bound function wrapper, we do not
however provide a ``__get__()`` method and thus this rebinding does not
however provide a `__get__()` method and thus this rebinding does not
occur. The result is that on the subsequent call, it all falls apart.
The solution therefore is that we need to add a ``__get__()`` method to the
The solution therefore is that we need to add a `__get__()` method to the
bound function wrapper which provides the ability to perform further
binding. We only want to do this where the instance was ``None``,
binding. We only want to do this where the instance was `None`,
indicating that the initial binding wasn't actually against an instance,
and where we are dealing with an instance method and not a class method or
static method.
@ -451,15 +459,15 @@ function and not the bound one. The simplest way of handling that is to
pass a reference to the original function wrapper to the bound function
wrapper and reach back into that to get the original wrapped function.
```
class bound_function_wrapper(object_proxy):
```python
class bound_function_wrapper(object_proxy):
def __init__(self, wrapped, instance, wrapper, binding, parent):
super(bound_function_wrapper, self).__init__(wrapped)
self.instance = instance
self.wrapper = wrapper
self.binding = binding
self.parent = parent
self.parent = parent
def __call__(self, *args, **kwargs):
if self.binding == 'function':
@ -471,16 +479,16 @@ class bound_function_wrapper(object_proxy):
return self.wrapper(self.wrapped, self.instance, args, kwargs)
else:
instance = getattr(self.wrapped, '__self__', None)
return self.wrapper(self.wrapped, instance, args, kwargs)
return self.wrapper(self.wrapped, instance, args, kwargs)
def __get__(self, instance, owner):
if self.instance is None and self.binding == 'function':
descriptor = self.parent.wrapped.__get__(instance, owner)
return bound_function_wrapper(descriptor, instance, self.wrapper,
self.binding, self.parent)
return self
return self
class function_wrapper(object_proxy):
class function_wrapper(object_proxy):
def __init__(self, wrapped, wrapper):
super(function_wrapper, self).__init__(wrapped)
@ -490,12 +498,12 @@ class function_wrapper(object_proxy):
elif isinstance(wrapped, staticmethod):
self.binding = 'staticmethod'
else:
self.binding = 'function'
self.binding = 'function'
def __get__(self, instance, owner):
wrapped = self.wrapped.__get__(instance, owner)
return bound_function_wrapper(wrapped, instance, self.wrapper,
self.binding, self)
self.binding, self)
def __call__(self, *args, **kwargs):
return self.wrapper(self.wrapped, None, args, kwargs)
@ -503,17 +511,17 @@ class function_wrapper(object_proxy):
Rerunning our most recent test once again we now get:
```
```pycon
>>> Class.function_rm = Class.function_im
>>> c.function_rm(1, 2)
INSTANCE <__main__.Class object at 0x105609790>
ARGS (1, 2)
ARGS (1, 2)
>>> Class.function_rm = Class.function_cm
>>> c.function_rm(1, 2)
INSTANCE <class '__main__.Class'>
ARGS (1, 2)
ARGS (1, 2)
>>> Class.function_rm = Class.function_sm
>>> c.function_rm(1, 2)
@ -531,32 +539,34 @@ in all cases so far our decorator has always been placed outside of the
existing decorators marking a method as either a class method or a static
method. What happens if we reverse the order?
```
```python
class Class(object):
@classmethod
@my_function_wrapper
def function_cm(self, a, b):
pass
pass
@staticmethod
@my_function_wrapper
def function_sm(a, b):
pass
pass
```
c = Class()
```pycon
>>> c = Class()
>>> c.function_cm(1,2)
INSTANCE None
ARGS (<class '__main__.Class'>, 1, 2)
ARGS (<class '__main__.Class'>, 1, 2)
>>> Class.function_cm(1, 2)
INSTANCE None
ARGS (<class '__main__.Class'>, 1, 2)
ARGS (<class '__main__.Class'>, 1, 2)
>>> c.function_sm(1,2)
INSTANCE None
ARGS (1, 2)
ARGS (1, 2)
>>> Class.function_sm(1, 2)
INSTANCE None