S3 Storage Vault HTTPs mode how to provide custom CA certificate

[Nj-kol]

When using S3 storage vault in Compute storage decoupled mode, how and where does one provide the .pem file or equilavent when trying to access in https mode

The doc does not mention anything - https://doris.apache.org/docs/dev/sql-manual/sql-statements/cluster-management/storage-management/CREATE-STORAGE-VAULT#s3-vault

CREATE STORAGE VAULT IF NOT EXISTS s3_demo_vault
PROPERTIES (
    "type" = "S3",                                      -- required
    "s3.endpoint" = "s3.us-east-1.amazonaws.com",       -- required
    "s3.access_key" = "xxxxxx",                         -- required,  Your S3 access key
    "s3.secret_key" = "xxxxxx",                         -- required,  Your S3 secret key
    "s3.region" = "us-east-1",                          -- required
    "s3.root.path" = "s3_demo_vault_prefix",            -- required
    "s3.bucket" = "xxxxxx",                             -- required,  Your S3 bucket name
    "provider" = "S3",                                  -- required
    "use_path_style" = "false"                          -- optional,  S3 recommended to set false
);

[gavinchou]

@Nj-kol what you mean is that you want to provide a certificate for the S3 client side when access S3?
do you use standard AWS S3 or other object storage service?

[Nj-kol]

@Nj-kol what you mean is that you want to provide a certificate for the S3 client side when access S3? do you use standard AWS S3 or other object storage service?

@gavinchou I managed to solve the issue, but I just wanted to outline the issue so that it can help the community at large in the future. Basically, the issue was that we have a custom vendor-backed S3 ( Cloudian ), which could only have been accessed over https. Now, since this is an on-prem private cloud installation, it was secured using our own private key.

So, Doris was unable to communicate with the remote S3 vault over HTTPS. The solution has bit nuanced since Metaservice and Backend are written in C++, and Frontend in Java. We had to make custom Docker images to import the public key.

For both Metaservice and Backend, we need to do the following:

RUN cp /opt/apache-doris/certs/Myorg.pem /usr/local/share/ca-certificates/myorg-ca.crt && \
    update-ca-certificates

And for Frontend, we need to do :

RUN keytool -import -noprompt -trustcacerts \
    -alias myorg-ca \
    -file /opt/apache-doris/certs/Myorg.pem \
    -keystore $JAVA_HOME/lib/security/cacerts \
    -storepass <SOME_PASS>

With this, the cert issue got solved.

[gavinchou]

Thank you for your feedback. That is a really useful usage example for other users use on-prem object storage service with HTTPS.

Let me try to conclude the issue.

The issue arises because you are using a self-signed Certificate Authority (CA) for your object storage service (Cloudian). As a result, Doris is unable to communicate with Cloudian due to the untrusted CA.

You installed your CA certificate into both the Linux system’s trust store and the JVM runtime environment to solve the trust issue.

---

And, one more question. How do your other product lines or clients access Cloudian?

Aside from adding the object storage CA certificate to the trust list, are there any alternative approaches you would suggest to resolve this issue? For example, would it be possible to specify which CAs to trust during Doris startup?

[Nj-kol]

Well most of our services are Java/Scala-based, for example Kafka Connect. So, it is sufficient in most cases for us to just import the pem file into Java's native keystore/truststore.

I don't think Doris (or any other tool) should automatically establish trust. However, I think this process of "how" to do it can be just documented