Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-46062

CTE reference node does not inherit the flag `isStreaming` from CTE definition node

    XMLWordPrintableJSON

Details

    Description

      Looks like this is a long standing bug.

      We figured out that CTE reference node would never set the isStreaming flag to true, regardless of the value for flag in CTE definition. The node cannot determine the right value of isStreaming flag by itself (likewise it cannot determine about resolution by itself) but it has no parameter in constructor, hence always takes the default (no children, so batch one).

      This may impact some rules which behaves differently depending on isStreaming flag. It would no longer be a problem once CTE reference is replaced with CTE definition at some point in "optimization phase", but all rules in analyzer and optimizer being triggered before the rule takes effect may be impacted.

      We probably couldn't sync the flag in real time, but we should sync the flag when we mark CTE reference to be "resolved". The rule `ResolveWithCTE` will be a good place to do that.

      Attachments

        Issue Links

          Activity

            People

              kabhwan Jungtaek Lim
              kabhwan Jungtaek Lim
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: