How to use Minio API in Golang

Introduction

Object Storages are versatile. We use Object Storages for almost every storage related applications now a days. One important reason is obviously the cheap price, other reasons can be their APIs available for almost every programming language, and so on. But what exactly is an Object Storage? Where can they be useful? Lets try to answer all such questions in this blog post.

Object Storage

What is Object Storage?

Object storage is basically a data storage architecture that manages data as objects, as opposed to other storage architectures like file systems, which manage data as a file hierarchy, or block storage, which manages data as blocks within sectors and tracks. Each object typically includes the data itself, a variable amount of metadata, and a globally unique identifier.

Where is Object Storage Useful?

Object storage is particularly useful in scenarios that require the storage and retrieval of large amounts of unstructured data. This includes use cases such as:

• Cloud Storage: Storing vast amounts of data in a scalable and cost-effective manner.
• Backup and Archiving: Long-term storage of data that is infrequently accessed but needs to be retained.
• Content Distribution: Delivering large media files, such as videos or images, to a wide audience.
• Big Data Analytics: Storing large datasets for processing and analysis.
• IoT Data Management: Handling large volumes of data generated by IoT devices.

The most popular Object Storage currently is the Amazon S3. It offers structured APIs, and is performant for efficient and bulk data storage and retrieval. However, Amazon S3 is relatively getting pricier every passing year. Or perhaps we can say, we now have much more economical options available in the market now a days. One such alternative is the MinIO Object Storage. It is an Open-source object storage option which is offered either as self-host or using the cloud verison as usual. Best part of all this is, MinIO APIs are fully compatible with the Amazon S3 APIs. So one need not much changes in the code for moving to MinIO, just in case this is desired.

MinIO

MinIO is an open-source object storage server compatible with the Amazon S3 cloud storage service. It offers a High performance, scalable and easy to use alternative to S3.

Note - 1

Setting up a MinIO is also described in a comprehensive blog post on our other portal which mainly deals with setups and installations.

Installation

To start with, lets get the latest version v7 of minio-go in our golang application using

go get github.com/minio/minio-go/v7

Now, in our golang application, let's create a directory for defining the s3 connector to access the minio storage.

We would need to have an already setup instance of Minio to continue further. If Minio instance is not setup, please follow this guide to install minio first.

Project Structure

We can structure our project something like this

├── configs │ └── config.go ├── go.mod ├── go.sum ├── main.go └── minio └── connector.go

As can be seen above, we have a configs directory to hold the config.go where we will read all the environment variables and supply our application with injected values.

Note - 2

This is a good practice to never store or hard code any sort of credentials in the code base, and always fetch the credentials from environment variables. Every now and then we see companies making such rookie mistakes, like hard coding API keys in the code which later gets leaked and API's are abused.

And we have a minio directory to hold the connector.go where we define all the code to access the minio instance. We will later demonstrate in main.go on how to use the minio connector we are defining here.

Minio Connector - Basic Setup

In the connector.go we can create a global variable to hold the client object. Every time we run the application, this variable is instantiated only once and reused for every subsequent calls to the minio instance.

So, we can define a variable s3client like this

// minio/connector.go package minio var s3Client *minio.Client

Now, lets define a utility function which would return an instantiated object of the s3client every time we call this function.

// minio/connector.go func GetMinio() *minio.Client { if s3Client == nil { // Instantiate the s3Client // ... } return s3Client }

Lets create another function which we can call to instantiate the s3client. This function will fetch the credentials from the config package (We will define this config.go right up next) and create a client object using minio.New().

// minio/connector.go package minio import ( "context" "fmt" config "userApplication/configs" "github.com/minio/minio-go/v7" "github.com/minio/minio-go/v7/pkg/credentials" ) var s3Client *minio.Client func InitMinio() { // Requests are always secure (HTTPS) by default. // Set secure=false to enable insecure (HTTP) access. // This boolean value is the last argument for New(). conn, err := minio.New(config.MinioEndpt, &minio.Options{ Creds: credentials.NewStaticV4(config.MinioAccessKey, config.MinioSecretKey, ""), Secure: true, }) if err != nil { fmt.Println(err) } found, err := conn.BucketExists(context.Background(), config.MinioBucket) if err != nil { fmt.Println(err) } if found { fmt.Println("Connection to Minio successful.") fmt.Println("Endpoint: " + config.MinioEndpt) fmt.Println("Bucket : " + config.MinioBucket) } s3Client = conn }

Also, in config.go we can use the os package to lookup the environment variables something like this

// configs/config.go package config import ( "os" _ "github.com/joho/godotenv/autoload" ) var ( // Minio MinioEndpt = getEnv("MINIO_ENDPOINT", "") MinioBucket = getEnv("MINIO_BUCKET", "") MinioAccessKey = getEnv("MINIO_ACCESS_KEY", "") MinioSecretKey = getEnv("MINIO_SECRET_KEY", "") // Similarly, We can fetch more env variables here, // for API keys, Database credentials etc ) func getEnv(key, fallback string) string { if value, ok := os.LookupEnv(key); ok { return value } return fallback }

Obviously, we also need a mechanism to load the env vars for the case while running the application / testing locally. To do so, we can simply define all the env variables in a file called .env kept at the root of the project, in the following format and install a package to autoload these values from the .env file

# Minio MINIO_ENDPOINT=<MINIO_HOST_ADDRESS> MINIO_BUCKET=<BUCKET_NAME> MINIO_ACCESS_KEY=<SOME_SECURE_ACCESS_KEY> MINIO_SECRET_KEY=<SOME_SECURE_SECRET_KEY>

we would also need to install the autoload package we used here using

go get github.com/joho/godotenv/autoload

So far the base structure of the minio connector is ready. We have a client object s3client using which we can get data from the bucket or write data to the bucket.

Lets take a look how we can read data.

Read Data from Bucket

We can read data from the bucket in two ways.

• Public URL
• Private URL

Public URL - What does this mean?

One use case can be when we need to have a public URL which can be used to access data multiple times, we use something called as presigned URLs. These presignedURLs allow users access to the data anytime they want. This is often used when running applications that allow users to upload/download publically accessible media files like images, videos, songs etc which the users intentionally wish to share with ease.

Note - 3

We can create a presigned URL valid for a maximum of 1 week.

Now, to create a presigned URL, we can create utility methods, as shown below

// minio/connector.go func HandlePanic() { r := recover() if r != nil { fmt.Println("RECOVER :", r) } } func GetPresignedURLFromMinio(objectname string) string { defer HandlePanic() reqParams := make(url.Values) // Gernerate presigned get object url. presignedURL, err := GetMinio().PresignedGetObject(context.Background(), config.MinioBucket, objectname, time.Second*24*60*60, reqParams) if err != nil { fmt.Println(err) return "" } return presignedURL.String() }

Note - 4

Note that we are using the utility method GetMinio() method which always return an initialized s3client object, to call other method like PresignedGetObject().

Private URL

We don't always need a publically accessible URL to access the object. Many times, all we need to do is simply download an object from the Object Storage and save it as a file in the local filesystem. This actually is the most prominent use case in object storages. This is often used when running applications that allow users to upload/download different formats of data on to object storage buckets. Formats include like pdf, xlsx, txt, jpg etc. These could be some reports, presentations, billings, or simply private/copyright restricted images.

Here's how we can do this.

// minio/connector.go func DownloadFileFromMinio(objectname string, filePath string) error { // Uncomment this in case the code goes into panic at any point of time // defer HandlePanic() // Download and save the object as a file in the local filesystem. err := GetMinio().FGetObject(context.Background(), config.MinioBucket, objectname, filePath, minio.GetObjectOptions{}) if err != nil { fmt.Println(err) return err } return nil }

Write Data to Bucket

Similar to Read, we can also write data to the bucket.

// minio/connector.go func UploadFileInMinio(objectname string, filePath string, contentType string) string { // Upload the test file with FPutObject info, err := GetMinio().FPutObject(context.Background(), config.MinioBucket, objectname, filePath, minio.PutObjectOptions{ContentType: contentType}) if err != nil { fmt.Println(err) return "" } fmt.Printf("Successfully uploaded %s of size %d\n", objectname, info.Size) return info.ETag }

Complete Connector Code

Complete code for the connector can be something like this

// minio/connector.go package minio import ( "context" "fmt" config "userApplication/configs" "net/url" "time" "github.com/minio/minio-go/v7" "github.com/minio/minio-go/v7/pkg/credentials" ) var s3Client *minio.Client func InitMinio() { // Requests are always secure (HTTPS) by default. // Set secure=false to enable insecure (HTTP) access. // This boolean value is the last argument for New(). conn, err := minio.New(config.MinioEndpt, &minio.Options{ Creds: credentials.NewStaticV4(config.MinioAccessKey, config.MinioSecretKey, ""), Secure: true, }) if err != nil { fmt.Println(err) } found, err := conn.BucketExists(context.Background(), config.MinioBucket) if err != nil { fmt.Println(err) } if found { fmt.Println("Connection to Minio successful.") fmt.Println("Endpoint: " + config.MinioEndpt) fmt.Println("Bucket : " + config.MinioBucket) } s3Client = conn } func GetMinio() *minio.Client { if s3Client == nil { InitMinio() } return s3Client } func HandlePanic() { r := recover() if r != nil { fmt.Println("RECOVER :", r) } } func DownloadFileFromMinio(objectname string, filePath string) error { defer HandlePanic() // Download and save the object as a file in the local filesystem. err := GetMinio().FGetObject(context.Background(), config.MinioBucket, objectname, filePath, minio.GetObjectOptions{}) if err != nil { fmt.Println(err) return err } return nil } func GetPresignedURLFromMinio(objectname string) string { defer HandlePanic() reqParams := make(url.Values) // Gernerate presigned get object url. presignedURL, err := GetMinio().PresignedGetObject(context.Background(), config.MinioBucket, objectname, time.Second*24*60*60, reqParams) if err != nil { fmt.Println(err) return "" } return presignedURL.String() } func UploadFileInMinio(objectname string, filePath string, contentType string) string { defer HandlePanic() // Upload the test file with FPutObject info, err := GetMinio().FPutObject(context.Background(), config.MinioBucket, objectname, filePath, minio.PutObjectOptions{ContentType: contentType}) if err != nil { fmt.Println(err) return "" } fmt.Printf("Successfully uploaded %s of size %d\n", objectname, info.Size) return info.ETag }

You can call these methods, to upload or download file or perhaps simply to get a PreSigned URL. In case there's something else you would want me to cover, feel free to post in the comments below.

Further Reading

Readers are encouraged to also take a look at the original documentation for the minio-go docs.

1. Minio-go Official Documentation
2. Minio-go Github

---

Feel free to write your opinions, questions or any errors you may have come across while reading this blog in the comments section below.

You can use the contact form as well.