Using Amazon's Java S3 client with Hetzner
This post is from JsPlumb, which is now in maintenance mode.
We recently completed a migration of infrastructure from AWS to Hetzner, saving quite a bit of cash in the process. It was quite straightforward for the most part - we created Cloud instances in Hetzner to replace our EC2 boxes, and span up a Postgres instance on a box of its own to replace RDS. We store packages for our private NPM repository on S3; Hetzner has Object Storage for that, so we migrated those, and we were pretty much done. But then a quick look at Hetzner's docs for Object Storage did not show any official Java libraries for interfacing with Object Storage, and in any case we had a whole persistence layer sitting on S3 that we were not particularly keen to rewrite!
From reading Hetzner's website we reached the conclusion that "S3 Compatible" probably meant we could access our buckets via the Amazon S3 client in Java, but how? Our original code looked like this:
import software.amazon.awssdk.regions.Region
import software.amazon.awssdk.services.s3.S3Client
S3Client s3Client = S3Client.builder().region(Region.MY_REGION).build()
To access an AWS API, you need to tell it an access key and a secret access key (which you've created in your AWS console). This code doesn't tell AWS anything about credentials, so how does it know what to use? The AWS client code tests various places looking for an access key and a secret access key. In our case we were using a credentials file in the ~/aws for development, and environment variables in production.
The full story
The single line of code above, due to the automatic credentials resolution process, is hiding a few key details. AWS actually needs to know 3 things when it makes an S3 request:
- What is the access key?
- What is the secret access key?
- What is the S3 endpoint?
The first two questions are answered by the credential resolution. The details for the S3 endpoint are filled in by AWS because we told it which Region to use. So to use a Hetzner bucket we need to be able to supply both the access key details and the Hetzner endpoint. We can do the first of these by supplying a CredentialsProvider, and the second by an endpointOverride.
Credentials Provider
We wrote our own credentials provider and a credentials model class:
import software.amazon.awssdk.auth.credentials.{AwsCredentials, AwsCredentialsProvider}
class JsPlumbCredentials implements AwsCredentials {
String accessKey;
String secretKey;
constructor(Configuration conf, String accessKeyProperty, String secretKeyProperty) {
this.accessKey = conf.get(accessKeyProperty);
this.secretKey = conf.get(secretKeyProperty);
}
String accessKeyId() {
return this.accessKey;
}
String secretAccessKey() {
return this.secretKey;
}
}
class JsPlumbCredentialsProvider implements AwsCredentialsProvider {
Configuration conf;
String accessKeyProperty;
String secretKeyProperty;
constructor(Configuration conf, String accessKeyProperty, String secretKeyProperty) {
this.conf = conf;
this.accessKeyProperty = accessKeyProperty;
this.secretKeyProperty = secretKeyProperty;
}
AwsCredentials resolveCredentials() {
return new JsPlumbCredentials(this.conf, this.accessKeyProperty, this.secretKeyProperty);
}
}
Configuration here is actually a class in the Play framework, but that's probably one implementation detail that will vary widely between implementations.
Now to tell AWS about this credentials provider:
S3Client s3Client = S3Client.builder()
.credentialsProvider(new JsPlumbCredentialsProvider(conf, "accessKeyProperty", "secretKeyProperty"))
.region(Region.MY_REGION)
.build()
Supplying the endpoint
The other thing that we need to tell AWS about is the endpoint for our Hetzner bucket:
S3Client s3Client = S3Client.builder()
.credentialsProvider(new JsPlumbCredentialsProvider(conf, "accessKeyProperty", "secretKeyProperty"))
.region(Region.MY_REGION)
.endpointOverride(new URI("https://nbg1.your-objectstorage.com"))
.build()
That URL is for Hetzner's Nuremberg storage. They have a couple of others.
And that's it. With those couple of changes we were able to retain our entire persistence API but swap out AWS for Hetzner under the hood.
Why are you still supplying region when connecting to Hetzner?
Perhaps you noticed in these last two code snippets that we're still supplying the region to AWS, and I said above that the reason we were supplying region was for it to figure out the S3 endpoint. Why am I still supplying it? Because computers, that's why! Because when I took it out the AWS client was unhappy and would not connect.
Try VisuallyJs
VisuallyJs offers an extensive list of starter apps, diagrams, charts and dashboards to quick start your development, in React, Angular, Vue, Svelte and Typescript/Javascript.
