Deferrable HttpSensor does not move to Triggerer when using response_check #47719
Conversation
# Conflicts: # providers/http/src/airflow/providers/http/sensors/http.py # providers/http/src/airflow/providers/http/triggers/http.py
|
@eladkal Still need to remove the pickling, probably the check should move so pickling won't be necessary anymore |
…d it also works in deferred mode
|
I think this one is ready |
| yield TriggerEvent( | ||
| { | ||
| "status": "success", | ||
| "response": base64.standard_b64encode(pickle.dumps(response)).decode("ascii"), |
There was a problem hiding this comment.
Can we use something else than pickle? Pickle is inherently insecure.
There was a problem hiding this comment.
I'll check, maybe use the DagBag and get code directly from the processed DAG file?
That's the trick I did to make the streamable XCom possible
There was a problem hiding this comment.
Maybe use same mechanism from PythonOperator? Maybe move that logic into common module (common-devel) so it an be reused accross providers (here standard and http)? WDYT?
_SERIALIZERS: dict[_SerializerTypeDef, Any] = {
"pickle": lazy_object_proxy.Proxy(_load_pickle),
"dill": lazy_object_proxy.Proxy(_load_dill),
"cloudpickle": lazy_object_proxy.Proxy(_load_cloudpickle),
}
There was a problem hiding this comment.
We should rather use our own serialization - and only one mechanism if possible
Closes: #40209
DAG Code:
Task going to Deferrable mode:
Output of the task:
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.