How to on-board new public cloud as back-end in OpenSDS multi-Cloud Project?

On board the new public Cloud as the back-end in OpenSDS will help users to perform all object storage-related operations such as “Create Bucket”, “Delete Bucket”, “Upload Object”, “Download Object”, and “Delete Object” directly from OpenSDS application. Also, the user will be able to migrate its bucket from any OpenSDS-supported public cloud or local storage to a newly onboarded cloud and vice-versa.

This blog will help you on how to add S3 compatible Cloud as a back-end in OpenSDS multi-cloud:

Add Cloud provider as object storage back-end in OpenSDS multi-cloud project:

Implement the below interfaces in the multi-cloud (Gelato) project for adding cloud as a back-end. More details about interfaces can be found here.

PUT(stream io.Reader, object *pb.Object, context context.Context) S3Error

GET(object *pb.Object, context context.Context) (io.ReadCloser, S3Error)

DELETE(object *pb.DeleteObjectInput, context context.Context) S3Error

GetObjectInfo(bucketName string, key string, context context.Context) (*pb.Object, S3Error)

InitMultipartUpload(object *pb.Object, context context.Context) (*pb.MultipartUpload, S3Error)

UploadPart(stream io.Reader, multipartUpload *pb.MultipartUpload, partNumber int64, upBytes int64, context context.Context) (*model.UploadPartResult, S3Error)

CompleteMultipartUpload(multipartUpload *pb.MultipartUpload, completeUpload *model.CompleteMultipartUpload, context context.Context) (*model.CompleteMultipartUploadResult, S3Error)

AbortMultipartUpload(multipartUpload *pb.MultipartUpload, context context.Context) S3Error

ListParts(listParts *pb.ListParts, context context.Context) (*model.ListPartsOutput, S3Error)

After successful implementation and integration of interfaces, perform the testing by registering the cloud as back-end with or without Keystone. Onboarded cloud provider name will appear in the “Type” drop-down. Refer to below screenshot.

Facilitate the Cloud which is added as a back-end for supporting migration in OpenSDS:

Implement Data Mover interfaces in multi-cloud (Gelato) project for newly added Cloud backend. You can refer implementation of the “Amazon S3 Data Mover” interface. It is called the migration routine.

Implement the below interfaces defined in MoveWorker.

• UploadObj(objKey string, destLoca *LocationInfo, buf []byte) error
• DownloadObj(objKey string, srcLoca *LocationInfo, buf []byte) (size int64, err error)
• DeleteObj(objKey string, loca *LocationInfo)
• MultiPartDownloadInit(srcLoca *LocationInfo) error
• DownloadRange(objKey string, srcLoca *LocationInfo, buf []byte, start int64, end int64) (size int64, err error)
• CompleteMultipartUpload(objKey string, destLoca *LocationInfo) error
• AbortMultipartUpload(objKey string, destLoca *LocationInfo) error
• UploadPart(objKey string, destLoca *LocationInfo, upBytes int64, buf []byte, partNumber int64, offset int64) error
• MultiPartUploadInit(objKey string, destLoca *LocationInfo)

After successful implementation and integration of interfaces, perform the testing by migrating Bucket from AWS S3, Microsoft Azure, and Huawei OBS to your onboarded Cloud and vice-versa.