[Disscuss] The precise of timestamp is limited to millisecond in carbondata, which is incompatiable with DB&Spark&Hive

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[Disscuss] The precise of timestamp is limited to millisecond in carbondata, which is incompatiable with DB&Spark&Hive

haomarch
The SimpleDateFormat is used to convert string and timestamp. But
SimpleDateFormat doesn't support nanoseconds and microsecond, which is is
incompatiable with DB&Spark&Hive.

To support nanoseconds and microsecond, we shall change SimpleDateFormat to
DateTimeFormatter.
Shall we involve this work´╝č



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: [Disscuss] The precise of timestamp is limited to millisecond in carbondata, which is incompatiable with DB&Spark&Hive

xubo245
+1, please consider compatable  for history data



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: [Disscuss] The precise of timestamp is limited to millisecond in carbondata, which is incompatiable with DB&Spark&Hive

Ajantha Bhat
+ 1,
as SimpleDateFormat doesn't support nanoseconds and microsecond.

Thanks,
Ajantha

On Tue, Jul 14, 2020 at 5:03 PM xubo245 <[hidden email]> wrote:

> +1, please consider compatable  for history data
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>
Reply | Threaded
Open this post in threaded view
|

Re: [Disscuss] The precise of timestamp is limited to millisecond in carbondata, which is incompatiable with DB&Spark&Hive

Zhangshunyu
In reply to this post by haomarch
+1



-----
My English name is Sunday
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
My English name is Sunday
Reply | Threaded
Open this post in threaded view
|

Re: [Disscuss] The precise of timestamp is limited to millisecond in carbondata, which is incompatiable with DB&Spark&Hive

ravipesala
Hi,

I think it is bigger than just changing to DateTimeFormatter. As of now,
carbon uses only 64 bit to store timestamp so it can accommodate till
milliseconds. In order to support till nanoseconds, we need to use 96
bits.  If you check spark-parquet it uses 96 bits to store timestamp. It
would be good if we also go in that direction to support till nanosecond
level. Thank you.

Regards,
Ravindra.

On Wed, 15 Jul 2020 at 14:50, Zhangshunyu <[hidden email]> wrote:

> +1
>
>
>
> -----
> My English name is Sunday
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>


--
Thanks & Regards,
Ravi
Reply | Threaded
Open this post in threaded view
|

Re: [Disscuss] The precise of timestamp is limited to millisecond in carbondata, which is incompatiable with DB&Spark&Hive

kumarvishal09
Hi,
I agree with Ravi, first we need to come up with solution how to store till
nano/mico second in carbon. Then changing the formatter will be useful for
us.

Regards
Kumar Vishal

On Wed, 15 Jul 2020 at 6:18 PM, Ravindra Pesala <[hidden email]>
wrote:

> Hi,
>
> I think it is bigger than just changing to DateTimeFormatter. As of now,
> carbon uses only 64 bit to store timestamp so it can accommodate till
> milliseconds. In order to support till nanoseconds, we need to use 96
> bits.  If you check spark-parquet it uses 96 bits to store timestamp. It
> would be good if we also go in that direction to support till nanosecond
> level. Thank you.
>
> Regards,
> Ravindra.
>
> On Wed, 15 Jul 2020 at 14:50, Zhangshunyu <[hidden email]> wrote:
>
> > +1
> >
> >
> >
> > -----
> > My English name is Sunday
> > --
> > Sent from:
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >
>
>
> --
> Thanks & Regards,
> Ravi
>
kumar vishal
Reply | Threaded
Open this post in threaded view
|

Re: [Disscuss] The precise of timestamp is limited to millisecond in carbondata, which is incompatiable with DB&Spark&Hive

David CaiQiang
In reply to this post by ravipesala
agree with Ravi



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Best Regards
David Cai