When a task that predates a chord in a chain was duplicated by Rabbitmq (for whatever reason),
the chord header id was not duplicated. This caused the chord header to have a different id.
This test ensures that the chord header's id preserves itself in face of such an edge case.
* Possible fix for uncaught rsa key error
* Raise ValueError when non-RSA certificate key is used
* Add certificate public key type check to Certificate.__init__. Public key must be of type RSAPublicKey, otherwise Certificate.verify method will fail
* Add unit tests for invalid key/certificate type
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update t/unit/security/__init__.py
* Fix linting
Co-authored-by: Oskar Vuola <oskar.vuola@labra.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* rename test class so it's executed
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add manager so task is executed
* fix test skipping
* make tests independent of prior results
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
In #6863 we discarded all jobs if synack isn't enabled for the pool.
This fixed a severe memory leak which occurs on connection restart.
Instead of going over each job and checking if we should discard it, we should clear the entire cache when there are no active writers.
If there are active writers, we should discard the jobs from the cache after we're done writing them since they also may remain on the cache forever.
* Make default worker state limits configurable
Previously, `REVOKES_MAX`, `REVOKE_EXPIRES`, `SUCCESSFUL_MAX` and
`SUCCESSFUL_EXPIRES` were hardcoded in `celery.worker.state`. This patch
introduces `CELERY_WORKER_` prefixed environment variables with the same
names that allow you to customize these values should you need to.
Fixes#3576.
* Add tests for configurable worker state limits
* Update sphinx-click from 4.2.0 to 4.3.0
* Update pre-commit from 2.19.0 to 2.20.0
* Pin elasticsearch to latest version 8.3.1
* Update zstandard from 0.17.0 to 0.18.0
* Update pytest-github-actions-annotate-failures from 0.1.6 to 0.1.7
* Update pycurl from 7.43.0.5 to 7.45.1
* elasticsearch<8.0
* pycurl==7.43.0.5
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* beat: Suppress banner output with the quiet option
This adds missing support for the quiet command line option
(`--quiet` or `-q`) for the celery beat command, which suppresses
banner and version information output.
Fixes#5836.
* beat: Add tests for the `--quiet` option
* Strip down the header-stamping PR to the basics.
* Serialize groups.
* Add groups to result backend meta data.
* Fix spelling mistake.
* Revert changes to canvas.py
* Revert changes to app/base.py
* Add stamping implementation to canvas.py
* Send task to AMQP with groups.
* Successfully pass single group to result.
* _freeze_gid dict merge fixed
* First draft of the visitor API.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* OptionsVisitor created
* Fixed canvas.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Added test for simple test for chord and fixed chord implementation
* Changed _IMMUTABLE_OPTIONS
* Fixed chord interface
* Fixed chord interface
* Fixed chord interface
* Fixed chord interface
* Fixed list order
* Fixed tests (stamp test and chord test), fixed order in groups
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fixed lint and elements
* Changed implementation of stamp API and fix lint
* Added documentation to Stamping API. Added chord with groups test
* Implemented stamping inside replace and added test for an implementation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Added test additonal tests for chord, improved coverage
* Added test additonal tests for chord, improved coverage
* Added test additonal tests for chord, improved coverage
* Splitted into subtests
* Group stamping rollback
* group.id is None fixed
* Added integration test
* Added integration test
* apply_async fixed
* Integration test and test_chord fixed
* Lint fixed
* chord freeze fixed
* Minor fixes.
* Chain apply_async fixed and tests fixed
* lint fixed
* Added integration test for chord
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* type -> isinstance
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Redo header stamping (#7341)
* _freeze_gid dict merge fixed
* OptionsVisitor created
* Fixed canvas.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Added test for simple test for chord and fixed chord implementation
* Changed _IMMUTABLE_OPTIONS
* Fixed chord interface
* Fixed chord interface
* Fixed chord interface
* Fixed chord interface
* Fixed list order
* Fixed tests (stamp test and chord test), fixed order in groups
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fixed lint and elements
* Changed implementation of stamp API and fix lint
* Added documentation to Stamping API. Added chord with groups test
* Implemented stamping inside replace and added test for an implementation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Added test additonal tests for chord, improved coverage
* Added test additonal tests for chord, improved coverage
* Added test additonal tests for chord, improved coverage
* Splitted into subtests
* Group stamping rollback
* group.id is None fixed
* Added integration test
* Added integration test
* apply_async fixed
* Integration test and test_chord fixed
* Lint fixed
* chord freeze fixed
* Minor fixes.
* Chain apply_async fixed and tests fixed
* lint fixed
* Added integration test for chord
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* type -> isinstance
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Omer Katz <omer.katz@omerkatz.com>
* Added stamping mechanism
* Manual stamping improved
* flake8 fixed
* Added subtests
* Add comma.
* Moved groups to stamps
* Fixed chord and added test for that
* Strip down the header-stamping PR to the basics.
* Serialize groups.
* Add groups to result backend meta data.
* Fix spelling mistake.
* Revert changes to canvas.py
* Revert changes to app/base.py
* Add stamping implementation to canvas.py
* Send task to AMQP with groups.
* Successfully pass single group to result.
* _freeze_gid dict merge fixed
* First draft of the visitor API.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* OptionsVisitor created
* Fixed canvas.py
* Added test for simple test for chord and fixed chord implementation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Changed _IMMUTABLE_OPTIONS
* Fixed chord interface
* Fixed chord interface
* Fixed chord interface
* Fixed chord interface
* Fixed list order
* Fixed tests (stamp test and chord test), fixed order in groups
* Fixed lint and elements
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Changed implementation of stamp API and fix lint
* Added documentation to Stamping API. Added chord with groups test
* Implemented stamping inside replace and added test for an implementation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Added test additonal tests for chord, improved coverage
* Added test additonal tests for chord, improved coverage
* Added test additonal tests for chord, improved coverage
* Splitted into subtests
* Group stamping rollback
* group.id is None fixed
* Added integration test
* Added integration test
* apply_async fixed
* Integration test and test_chord fixed
* Lint fixed
* chord freeze fixed
* Minor fixes.
* Chain apply_async fixed and tests fixed
* lint fixed
* Added integration test for chord
* type -> isinstance
* Added stamping mechanism
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Manual stamping improved
* fail_ci_if_error uncommented
* flake8 fixed
* Added subtests
* Changes
* Add comma.
* Fixed chord and added test for that
* canvas.py fixed
* Test chord.py fixed
* Fixed stamped_headers
* collections import fixed
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* collections import fixed
* Update celery/backends/base.py
Co-authored-by: Omer Katz <omer.katz@omerkatz.com>
* ampq.py fixed
* Refrain from using deprecated import path.
* Fix test_complex_chain regression.
Whenever we stamp a group we need to freeze it first if it wasn't already frozen.
Somewhere along the line, the group id changed because we were freezing twice.
This commit places the stamping operation after preparing the chain's steps which fixes the problem somehow.
We don't know why yet.
* Fixed integration tests
* Fixed integration tests
* Fixed integration tests
* Fixed integration tests
* Fixed issues with maybe_list. Add documentation
* Fixed potential issue with integration tests
* Fixed issues with _regen
* Fixed issues with _regen
* Fixed test_generator issues
* Fixed _regen stamping
* Fixed _regen stamping
* Fixed TimeOut issue
* Fixed TimeOut issue
* Fixed TimeOut issue
* Update docs/userguide/canvas.rst
Co-authored-by: Omer Katz <omer.katz@omerkatz.com>
* Fixed Couchbase
* Better stamping intro
* New GroupVisitor example
* Adjust documentation.
Co-authored-by: Naomi Elstein <naomi.els@omerkatz.com>
Co-authored-by: Omer Katz <omer.katz@omerkatz.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
* let group, chain and chunks create the subclassses when recreating from dict
* add tests for subclassing workflow primitives
* force the creation of the proper chain type when subclassing chain
Co-authored-by: Eric Yen <eric@ericyen.com>
* Remove deprecated methods in `celery.local.Proxy`
* Collapse conditionals for readability
* Remove unused parameter `uuid`
* Remove unused import `ClusterOptions`
* Remove dangerous mutable default argument
Continues work from #5478
* Remove always `None` and unused global variable
* Remove unreachable `elif` block
* Consolidate import statements
* Add missing parameter to `os._exit()`
* Add missing assert statement
* Remove unused global `WindowsError`
* Use `mkstemp` instead of deprecated `mktemp`
* No need for `for..else` constructs in loops that don't break
In these cases where the loop returns or raises instead of breaking, it
is simpler to just put the code that runs after the loop completes right
after the loop instead.
* Use the previously unused parameter `compat_modules`
Previously this parameter was always overwritten by the value of
`COMPAT_MODULES.get(name, ())`, which was very likely unintentional.
* Remove unused local variable `tz`
* Make `assert_received` actually check for `is_received`
Previously, it called `is_accepted`, which was likely a copy-paste
mistake from the `assert_accepted` method.
* Use previously unused `args` and `kwargs` params
Unlike other backends' `__reduce__` methods, the one from `RedisBackend`
simply overwrites `args` and `kwargs` instead of adding to them. This
change makes it more in line with other backends.
* Update celery/backends/filesystem.py
Co-authored-by: Gabriel Soldani <1268700+gabrielsoldani@users.noreply.github.com>
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* Reduce prefetch count on restart and gradually restore it.
* Rename.
* Ensure we always eventually increase the QoS.
* Ensure _maximum_prefetch_restored isn't racy.
* Set default num_processes to 2.
* Add unit test.
* Celery set prefetch count on restart (#7390)
* Added test for create_task_handler
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Changed consumer.py
* Fixed celery consumer implementation style
* Improved test_create_task_handler. Used task_message_from_sig instread of manual mocking
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Celery set prefetch count on restart (#7403)
* Added test for create_task_handler
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Changed consumer.py
* Fixed celery consumer implementation style
* Improved test_create_task_handler. Used task_message_from_sig instread of manual mocking
* Fixed test_worker.py
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add a log message to notify users we're reducing the prefetch count.
* Notify user when normal operations are resumed.
* Document broker reconnection behaviour
* Marked which feature was added in which version.
Co-authored-by: dobosevych <dobosevych@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
When task_always_eager is enabled, the backend issues a warning when trying to get task data, This cancels this warning in case task_store_eager_result is enabled as well.
* Update sphinx-click from 4.1.0 to 4.2.0
* Pin elasticsearch to latest version 8.2.3
* Update pycurl from 7.43.0.5 to 7.45.1
* pycurl==7.43.0.5
* elasticsearch<8.0
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* docs: ensure consistency in `Backends and Brokers` descriptions
* docs: replace semicolon with colon in `First Steps with Celery`
* docs: replace backtick with apostrophe in `Frequently Asked Questions`
* Pin billiard to latest version 4.0.0
* Pin sphinx-click to latest version 4.1.0
* Pin pytest-celery to latest version 0.0.0
* Pin pytest-celery to latest version 0.0.0
* Pin pytest-click to latest version 1.1.0
* Pin mypy to latest version 0.961
* Update pre-commit from 2.18.1 to 2.19.0
* Update msgpack from 1.0.3 to 1.0.4
* Pin python-consul2 to latest version 0.1.5
* Update pycouchdb from 1.14.1 to 1.14.2
* Pin elasticsearch to latest version 8.2.2
* Pin zstandard to latest version 0.17.0
* Pin pydocstyle to latest version 6.1.1
* Pin bumpversion to latest version 0.6.0
* Pin pytest-cov to latest version 3.0.0
* Pin pytest-github-actions-annotate-failures to latest version 0.1.6
* Update pycurl from 7.43.0.5 to 7.45.1
* pin ElS
* pycurl==7.43.0.5
* billiard>=3.6.4.0,<5.0
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* Include `dont_autoretry_for` option in tasks.
* Fixed issue with failing tests
* Fixed flake8 issue
* Change version added
Co-authored-by: Manuel Vázquez Acosta <manuel@merchise.org>
* tFixed celery task expiration check, expires value can be ISO8601 for task retry
* Fixed tests and migrate them to another branch
* Fixed tests and migrate them to another branch
Co-authored-by: Luboš Mátl <matllubos@gmail.com>
Tomer Nosrati