[DISCUSSION] Support alter schema for complex types

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSSION] Support alter schema for complex types

akshay_nuthala
Hi everyone,

Currently carbon only supports altering schema for primitive types.
We can extend it to complex columns too -
1) alter add complex column(array/struct)
2) alter drop complex column(array/struct)
3) alter change datatype/rename complex column(array/struct)

Here phase-1 focuses only on non-nested complex columns.
Phase-2 will later target nested ones.

Please let me know your valid inputs about the same.
Following is the link to the design document.

https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit

Thanks,
Akshay




--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support alter schema for complex types

Indhumathi
+1. It is an important feature needs to be supported.

Since, you have mentioned alter drop struct child column, are you gonna
handle below scenarios also in phase 1?
1. support rename/change struct primitive child members ?
2. add new primitive child members in an existing struct ?

If so, please update design for those DDL commands.

Regards,
Indhumathi M



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support alter schema for complex types

akashrn5
In reply to this post by akshay_nuthala
Hi,

+1 for feature. Thanks for proposing it as now most of the use case from
user perspective involves complex columns.

I have reviewed the doc and given comments, please work on it, then can be
reviewed again.

Regards,
Akash R



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support alter schema for complex types

akshay_nuthala
In reply to this post by Indhumathi
No, these and other nested level operations will be taken care in the next
phase.



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support alter schema for complex types

kunalkapoor
+1

On Fri, Mar 26, 2021 at 6:19 PM akshay_nuthala <[hidden email]>
wrote:

> No, these and other nested level operations will be taken care in the next
> phase.
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support alter schema for complex types

Ajantha Bhat
Hi Akshay,
The mail description and document content are not matching. For
single-level struct also document says cannot support.
So, please list down all the work that need to be done in points and
then divide which is supported in phase1 and which is supported in phase 2
clearly in the summary section of the document.

Also in the query flow, after adding the column, for previously loaded
segments what will be the output NULL or empty complex type ?
you can refer hive behavior for this.  Hope schema evolution (column drift)
also intact with complex column support.

Thanks,
Ajantha

On Tue, Mar 30, 2021 at 11:18 AM Kunal Kapoor <[hidden email]>
wrote:

> +1
>
> On Fri, Mar 26, 2021 at 6:19 PM akshay_nuthala <[hidden email]>
> wrote:
>
> > No, these and other nested level operations will be taken care in the
> next
> > phase.
> >
> >
> >
> > --
> > Sent from:
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support alter schema for complex types

Mahesh Raju Somalaraju
In reply to this post by akshay_nuthala
Hi,

+1 For the feature. Agreed with Ajantha.

Thanks & Regards
Mahesh Raju Somalaraju

On Thu, Mar 18, 2021 at 10:00 PM akshay_nuthala <[hidden email]>
wrote:

> Hi everyone,
>
> Currently carbon only supports altering schema for primitive types.
> We can extend it to complex columns too -
> 1) alter add complex column(array/struct)
> 2) alter drop complex column(array/struct)
> 3) alter change datatype/rename complex column(array/struct)
>
> Here phase-1 focuses only on non-nested complex columns.
> Phase-2 will later target nested ones.
>
> Please let me know your valid inputs about the same.
> Following is the link to the design document.
>
>
> https://docs.google.com/document/d/1DhhkVXM8rMvOuKDZeccJpFEfO3VkA9C0c7JHCV88NXI/edit
>
> Thanks,
> Akshay
>
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] Support alter schema for complex types

akshay_nuthala
In reply to this post by akashrn5