[C++] Cannot locate UTC timezone error on Windows machine

[alinaliBQ]

Describe the bug, including details regarding any error messages, version, and platform.

Our team discovered that when function call LocateZone("UTC") is reached on Windows machine under certain conditions, it produces error:

Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\<username>\Downloads\tzdata"

I am not familiar with the LocateZone function. Brief search shows tzdata is a python package and my machine actually has it, but I don't think the error has to do with tzdata package.

I am seeing this Timezone database not found error only when we are trying to convert the timestamp data to a char value, and the arrow::TimestampType has null timestamp data with UTC timezone.

Simplified function call that triggered the error:

// set up original array and cast options
const std::shared_ptr<arrow::Array>& original_array; // null timestamp data with UTC timezone
arrow::compute::CastOptions cast_options; // cast from timestamp to char type
return arrow::compute::CallFunction("cast", {original_array}, &cast_options);

The call stack:

arrow.dll!arrow::compute::internal::LocateZone(const std::string & timezone) Line 47
	at C:\Users\AlinaLi\Documents\Arrow-Flight-SQL\bqarrow\cpp\src\arrow\compute\kernels\temporal_internal.h(47)
arrow.dll!arrow::compute::internal::`anonymous namespace'::TemporalToStringCastFunctor<arrow::StringType,arrow::TimestampType>::ConvertZoned<std::chrono::duration<__int64,std::ratio<1,1000>>>(const arrow::ArraySpan & input, const std::string & timezone, arrow::StringBuilder * builder) Line 199
	at C:\Users\AlinaLi\Documents\Arrow-Flight-SQL\bqarrow\cpp\src\arrow\compute\kernels\scalar_cast_string.cc(199)
arrow.dll!arrow::compute::internal::`anonymous namespace'::TemporalToStringCastFunctor<arrow::StringType,arrow::TimestampType>::Exec(arrow::compute::KernelContext * ctx, const arrow::compute::ExecSpan & batch, arrow::compute::ExecResult * out) Line 171
	at C:\Users\AlinaLi\Documents\Arrow-Flight-SQL\bqarrow\cpp\src\arrow\compute\kernels\scalar_cast_string.cc(171)
arrow.dll!arrow::compute::detail::`anonymous namespace'::ScalarExecutor::ExecuteNonSpans(arrow::compute::detail::ExecListener * listener) Line 920
	at C:\Users\AlinaLi\Documents\Arrow-Flight-SQL\bqarrow\cpp\src\arrow\compute\exec.cc(920)
arrow.dll!arrow::compute::detail::`anonymous namespace'::ScalarExecutor::Execute(const arrow::compute::ExecBatch & batch, arrow::compute::detail::ExecListener * listener) Line 810
	at C:\Users\AlinaLi\Documents\Arrow-Flight-SQL\bqarrow\cpp\src\arrow\compute\exec.cc(810)
arrow.dll!arrow::compute::detail::FunctionExecutorImpl::Execute(const std::vector<arrow::Datum,std::allocator<arrow::Datum>> & args, __int64 passed_length) Line 278
	at C:\Users\AlinaLi\Documents\Arrow-Flight-SQL\bqarrow\cpp\src\arrow\compute\function.cc(278)
arrow.dll!arrow::compute::`anonymous namespace'::ExecuteInternal(const arrow::compute::Function & func, std::vector<arrow::Datum,std::allocator<arrow::Datum>> args, __int64 passed_length, const arrow::compute::FunctionOptions * options, arrow::compute::ExecContext * ctx) Line 343
	at C:\Users\AlinaLi\Documents\Arrow-Flight-SQL\bqarrow\cpp\src\arrow\compute\function.cc(343)
arrow.dll!arrow::compute::Function::Execute(const std::vector<arrow::Datum,std::allocator<arrow::Datum>> & args, const arrow::compute::FunctionOptions * options, arrow::compute::ExecContext * ctx) Line 350
	at C:\Users\AlinaLi\Documents\Arrow-Flight-SQL\bqarrow\cpp\src\arrow\compute\function.cc(350)
arrow.dll!arrow::compute::internal::`anonymous namespace'::CastMetaFunction::ExecuteImpl(const std::vector<arrow::Datum,std::allocator<arrow::Datum>> & args, const arrow::compute::FunctionOptions * options, arrow::compute::ExecContext * ctx) Line 125
	at C:\Users\AlinaLi\Documents\Arrow-Flight-SQL\bqarrow\cpp\src\arrow\compute\cast.cc(125)
arrow.dll!arrow::compute::MetaFunction::Execute(const std::vector<arrow::Datum,std::allocator<arrow::Datum>> & args, const arrow::compute::FunctionOptions * options, arrow::compute::ExecContext * ctx) Line 484
	at C:\Users\AlinaLi\Documents\Arrow-Flight-SQL\bqarrow\cpp\src\arrow\compute\function.cc(484)
arrow.dll!arrow::compute::CallFunction(const std::string & func_name, const std::vector<arrow::Datum,std::allocator<arrow::Datum>> & args, const arrow::compute::FunctionOptions * options, arrow::compute::ExecContext * ctx) Line 1369
	at C:\Users\AlinaLi\Documents\Arrow-Flight-SQL\bqarrow\cpp\src\arrow\compute\exec.cc(1369)
arrow_flight_sql_odbc.dll!driver::flight_sql::GetConverter::__l27::<lambda_7>::operator()(const std::shared_ptr<arrow::Array> & original_array) Line 1036

Any insights would be helpful, thanks!

Component(s)

C++, FlightRPC

[alinaliBQ]

Hi @lidavidm @kou Do you know who should be tagged look into this issue?

[lidavidm]

AIUI, look at https://howardhinnant.github.io/date/tz.html#Installation to get the time zone database installed. Also #31472 I think is related/this is a duplicate of that, possibly.

[kou]

Our CI uses https://github.com/apache/arrow/blob/main/ci/scripts/download_tz_database.sh to download the time zone database.

[alinaliBQ]

Thanks for providing the info to download the time zone database. Is there any workaround for the call arrow::compute::CallFunction("cast", ...) for Timestamp conversion to work on end user's machines without having to download the time zone database?

Also, I am a bit confused; I am not sure if the tzdata time zone database is only meant to be needed for testing purpose in a development environment, or is tzdata needed for all users that want to use Arrow compute conversion call?

For reference, the commit that implements compute timestamp conversion: f3f4423

[kou]

If you need to support only UTC, we may be able embed UTC time zone information to our code base.

If you need to support time zones except UTC, we need the time zone database.

Also, I am a bit confused; I am not sure if the tzdata time zone database is only meant to be needed for testing purpose in a development environment, or is tzdata needed for all users that want to use Arrow compute conversion call?

The latter: tzdata is needed for all users that want to use Arrow compute conversion cal

[alinaliBQ]

Hi Kou, thanks for confirming that tzdata is needed. I think if UTC is supported then it would help in my case. And maybe there is a simpler way that I can suggest without bringing in the time zone database:

Here is the current code

arrow/cpp/src/arrow/compute/kernels/scalar_cast_string.cc

Lines 194 to 202 in 7df396e

	static Status ConvertZoned(const ArraySpan& input, const std::string& timezone,
	BuilderType* builder) {
	static const std::string kFormatString = "%Y-%m-%d %H:%M:%S%z";
	static const std::string kUtcFormatString = "%Y-%m-%d %H:%M:%SZ";
	DCHECK(!timezone.empty());
	ARROW_ASSIGN_OR_RAISE(const time_zone* tz, LocateZone(timezone));
	ARROW_ASSIGN_OR_RAISE(std::locale locale, GetLocale("C"));
	TimestampFormatter<Duration> formatter{
	timezone == "UTC" ? kUtcFormatString : kFormatString, tz, locale};

Is it possible to change it to look like the following pseudo-code (with my imperfect draft syntax):

TimestampFormatter<Duration> formatter;
if ( timezone == "UTC") {
   formatter = { kUtcFormatString };
} else {
   ARROW_ASSIGN_OR_RAISE(const time_zone* tz, LocateZone(timezone)); 
   ARROW_ASSIGN_OR_RAISE(std::locale locale, GetLocale("C")); 
   formatter = { kFormatString, tz, locale}; 
}

this way since UTC covers many use cases, we can omit the LocateZone call that isn't needed for when timezone is UTC.

[kou]

Let's try it. (It seems that we need to create tz even when we use UTC.)

[alinaliBQ]

Great, let me try; I will post the PR for the change here when it gets created

[alinaliBQ]

I have tried the changes on my local end, and you were right. tz is needed.