Connect Trino to data on local disk

[ajeyabsf]

I am new to Trino and want to do some benchmarking. I am using version 472, and trying to work with Trino using TPC-DS data stored on local disk (I currently don't want to try with tpcds connector which generates data on the fly). The data is in parquet format. I am trying to use the hive connector ( I am unsure if this is the option to choose).

The data is stored here.

~/tpcds_local/tpcds-data$ ls
call_center catalog_sales customer_demographics income_band promotion store time_dim web_returns
catalog_page customer date_dim inventory reason store_returns warehouse web_sales
catalog_returns customer_address household_demographics item ship_mode store_sales web_page web_site

In hive.properties, I have added this info

connector.name=hive
hive.metastore=file
hive.metastore.catalog.dir=local:///home/user/tpcds_local/tpcds-data
fs.native-local.enabled=true
hive.non-managed-table-creates-enabled=true
hive.non-managed-table-writes-enabled=true

When I create the table using

...
WITH (
external_location = 'file:///home/user/tpcds_local/tpcds-data/customer',
format = 'PARQUET'
);

, there is no data being populated.

How do I populate the data from the local filesystem?
Request some help on this

[fywxin]

so do i, help us please~

[LuckyCaptain-go]

Solution for Using TPC-DS Parquet Data with Trino (v472) via Hive Connector

1. Fix Configuration (etc/catalog/hive.properties)

connector.name=hive
hive.metastore=file
# Use a SEPARATE empty directory for metastore (not your data directory)
hive.metastore.catalog.dir=file:///home/user/tpcds_local/metastore
fs.native-local.enabled=true
hive.non-managed-table-creates-enabled=true

Critical Steps:

# Create new metastore directory
mkdir -p ~/tpcds_local/metastore

# Set permissions (Trino needs read/write access)
chmod -R 755 ~/tpcds_local

2. Create Database

CREATE SCHEMA hive.tpcds
WITH (location = 'file:///home/user/tpcds_local/tpcds-data');

3. Create Tables with Correct Schema

For each table, specify:

• Exact column names and types
• Correct Parquet file path
• PARQUET format

Example for customer table:

USE hive.tpcds;

CREATE TABLE customer (
  c_customer_sk             BIGINT,
  c_customer_id             VARCHAR(16),
  c_current_cdemo_sk        BIGINT,
  c_current_hdemo_sk        BIGINT,
  c_current_addr_sk         BIGINT,
  c_first_shipto_date_sk    BIGINT,
  c_first_sales_date_sk     BIGINT,
  c_salutation              VARCHAR(10),
  c_first_name              VARCHAR(20),
  c_last_name               VARCHAR(30),
  c_preferred_cust_flag     VARCHAR(1),
  c_birth_day               INTEGER,
  c_birth_month             INTEGER,
  c_birth_year              INTEGER,
  c_birth_country           VARCHAR(20),
  c_login                   VARCHAR(13),
  c_email_address           VARCHAR(50),
  c_last_review_date        VARCHAR(10)
)
WITH (
  format = 'PARQUET',
  external_location = 'file:///home/user/tpcds_local/tpcds-data/customer'
);

4. Verify Data Access

-- Check table list
SHOW TABLES FROM hive.tpcds;

-- Query sample data
SELECT * FROM customer LIMIT 5;

5. Troubleshooting Checklist

1. File Structure: Ensure each table directory contains actual Parquet files:

   ls ~/tpcds_local/tpcds-data/customer
   # Should show: part-00000.parquet, part-00001.parquet, etc.

2. Schema Mismatch: Verify column definitions match Parquet schema using:

   parquet-tools schema ~/tpcds_local/tpcds-data/customer/part-00000.parquet

3. Trino Logs: Check for errors:

   tail -f /var/log/trino/server.log

4. Permissions: Ensure Trino process has read access:

   chmod -R o+rX ~/tpcds_local

6. Automation Script

Generate DDL for all tables:

#!/bin/bash
TABLES=(call_center catalog_page catalog_returns catalog_sales customer customer_address customer_demographics date_dim household_demographics income_band inventory item promotion reason ship_mode store store_returns store_sales time_dim warehouse web_page web_returns web_sales web_site)

for table in "${TABLES[@]}"; do
  echo "CREATE TABLE $table (...) WITH (format='PARQUET', external_location='file:///home/user/tpcds_local/tpcds-data/$table');"
done > create_tables.sql

Note: Replace (...) with actual schema from your Parquet files.

Key Points:

• Use file:// prefix for local paths
• Metastore directory MUST be separate from data directory
• Table schemas MUST match Parquet file schemas
• Restart Trino after configuration changes:

  trino-service restart

For TPC-DS schema reference, see: TPC-DS Table Definitions