Celery task 执行报错 TypeError: Object of type set is not JSON serializable 问题分析处理

情况描述：

定义了新的shared_task推送到生产环境后，发现无法执行，会报错set对象无法序列化，报错内容如下：

Traceback (most recent call last):File "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errorsyieldFile "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 221, in dumpspayload = encoder(data)File "/tmp/venv/lib64/python3.6/site-packages/kombu/utils/json.py", line 70, in dumps**dict(default_kwargs, **kwargs))File "/tmp/venv/lib64/python3.6/site-packages/simplejson/__init__.py", line 412, in dumps**kw).encode(obj)File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 296, in encodechunks = self.iterencode(o, _one_shot=True)File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 378, in iterencodereturn _iterencode(o, 0)File "/tmp/venv/lib64/python3.6/site-packages/kombu/utils/json.py", line 59, in defaultreturn super(JSONEncoder, self).default(o)File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 273, in defaulto.__class__.__name__)
TypeError: Object of type set is not JSON serializableDuring handling of the above exception, another exception occurred:Traceback (most recent call last):File "/tmp/venv/lib64/python3.6/site-packages/celery/app/trace.py", line 449, in trace_taskuuid, retval, task_request, publish_result,File "/tmp/venv/lib64/python3.6/site-packages/celery/backends/base.py", line 149, in mark_as_doneself.store_result(task_id, result, state, request=request)File "/tmp/venv/lib64/python3.6/site-packages/celery/backends/base.py", line 342, in store_resultrequest=request, **kwargs)File "/tmp/venv/lib64/python3.6/site-packages/django_celery_results/backends/database.py", line 19, in _store_resultcontent_type, content_encoding, result = self.encode_content(result)File "/tmp/venv/lib64/python3.6/site-packages/django_celery_results/backends/database.py", line 52, in encode_contentcontent_type, content_encoding, content = self._encode(data)File "/tmp/venv/lib64/python3.6/site-packages/celery/backends/base.py", line 297, in _encodereturn dumps(data, serializer=self.serializer)File "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 221, in dumpspayload = encoder(data)File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__self.gen.throw(type, value, traceback)File "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 54, in _reraise_errorsreraise(wrapper, wrapper(exc), sys.exc_info()[2])File "/tmp/venv/lib64/python3.6/site-packages/vine/five.py", line 194, in reraiseraise value.with_traceback(tb)File "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errorsyieldFile "/tmp/venv/lib64/python3.6/site-packages/kombu/serialization.py", line 221, in dumpspayload = encoder(data)File "/tmp/venv/lib64/python3.6/site-packages/kombu/utils/json.py", line 70, in dumps**dict(default_kwargs, **kwargs))File "/tmp/venv/lib64/python3.6/site-packages/simplejson/__init__.py", line 412, in dumps**kw).encode(obj)File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 296, in encodechunks = self.iterencode(o, _one_shot=True)File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 378, in iterencodereturn _iterencode(o, 0)File "/tmp/venv/lib64/python3.6/site-packages/kombu/utils/json.py", line 59, in defaultreturn super(JSONEncoder, self).default(o)File "/tmp/venv/lib64/python3.6/site-packages/simplejson/encoder.py", line 273, in defaulto.__class__.__name__)
kombu.exceptions.EncodeError: Object of type set is not JSON serializable

初步分析

a.考虑是task调用过程中，参数传输过程异常，导致参数无法序列化，因为set 就是{1,2} 这种格式的，命令行验证没问题
b.考虑是其他模块有异常，发现其他模块的定时任务能正常执行
c.在测试环境能完整复现，但是由于celery 的调用无法调试，无法得到有效的反馈
d.命令行直接执行task函数不会报错，所以函数逻辑应该没有问题
e.最终发现是结果集里面存在set集合，导致后续结果传输的过程中无法解析成json，至此问题确定

解决办法：

将set转换成其他数据结构

总结：

a.对于原先测试验证能通过，修改后无法验证通过的，要先找到变更的内容，修改后要验证
b.这次花费时间较长是因为认为原先验证过，认为后续小的修改没有影响，导致思路不对，修改代码要验证，不要过于自信
c.对于celery这种无法调试的调用过程，复现存在难度，那就需要从函数逻辑，返回结果里面找答案
d.可能会转换成json格式的数据，一定不能包含set集合
e.其他：@shared_task 注解的函数，会加载到内存里面，需要重启Celery才会重新加载