aistore/api/README.md at master · ryan-beisner/aistore

AIStore Client API

Overview

The AIStore Client API package provides wrappers for core AIStore RESTful operations. The api package can be imported by other projects for quickly integrating AIStore functionality with minimal imports of other supporting packages.

Types of Operations

The APIs provided are separated into different levels of granularity:

Cluster,
Daemon,
Bucket,
Object.

Cluster

GetClusterMap

Retrieves AIStore's cluster map (Smap).

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)

Return

A copy of type cluster.Smap containing a map of targets, a map of proxies, proxies non-eligible for primary, the current primary proxy, and the version of the cluster map

Error from AIStore in completing the request

SetPrimaryProxy

Given a daemonID, it sets that proxy as the primary proxy of the cluster

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
newPrimaryID	string	DaemonID of the new primary proxy

Return

Error from AIStore in completing the request

SetClusterConfig

Given key-value pairs of configuration parameters, this operation sets the cluster-wide configuration accordingly. Setting cluster-wide configuration requires sending the request to a proxy

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
nvs	key-values	Map of key-value pairs of configuration parameters to be set

Return

Error from AIStore in completing the request

RegisterNode

Registers an existing node to the clustermap.

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
nodeInfo	*cluster.Snode	Pointer to a cluster.Snode struct containing details of the new node

Return

Error from AIStore in completing the request

UnregisterNode

Unregisters an existing node from the clustermap.

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
unregisterSID	string	DaemonID of the node to be unregistered

Return

Error from AIStore in completing the request

Daemon

GetMountpaths

Given the direct public URL of a target, GetMountpaths returns its mountpaths

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
targetURL	string	URL of the target node

Return

cmn.MountpathList structure comprising available and disabled mountpaths of a specific target

Error from AIStore in completing the request

AddMountpath

Given a target and a mountpath, AddMountpath adds that mountpath to the specified target

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
targetURL	string	URL of the target node
mountPath	string	Mountpath to be added to a target

Return

Error from AIStore in completing the request

RemoveMountpath

Given a target and a mountpath, RemoveMountpath removes that mountpath from the specified target

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
targetURL	string	URL of the target node
mountPath	string	Mountpath to be removed from a target

Return

Error from AIStore in completing the request

EnableMountpath

Given a target and a mountpath, EnableMountpath enables that mountpath on the specified target

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
targetURL	string	URL of the target node
mountPath	string	Mountpath to be enabled on a target

Return

Error from AIStore in completing the request

DisableMountpath

Given a target and a mountpath, DisableMountpath disables that mountpath on the specified target

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
targetURL	string	URL of the target node
mountPath	string	Mountpath to be disabled on a target

Return

Error from AIStore in completing the request

GetDaemonConfig

Given the URL of a daemon, GetDaemonConfig returns the corresponding daemon's configuration settings.

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
daemonURL	string	URL of the daemon (proxy/gateway node or target node)

Return

cmn.Config structure containing all the node's configuration settings (the node is specified by its daemonURL)

Error from AIStore in completing the request

SetDaemonConfig

Given key-value pairs of configuration parameters, SetDaemonConfig sets the configuration accordingly for a specific daemon

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
daemonURL	string	URL of the daemon (proxy/gateway node or target node)
nvs	key-values	Map of key-value pairs of configuration parameters to be set

Return

Error from AIStore in completing the request

Bucket

HeadBucket

Given a bucket name, returns its properties

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the bucket
query	url.Values	Optional URL query values such as `?provider`

Return

cmn.BucketProps structure containing the bucket's properties

Error from AIStore in completing the request

ListBuckets

Given the url of an AIS gateway, ListBuckets returns the names of all existing local and remote buckets

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
provider	string	"ais" or one of the supported backend providers: "ais", "gcp", "aws", "azure", "http", "hdfs". If the provider value is empty, returns all bucket names.

Return

Two lists: one for the names of ais buckets, and the other for the names of remote buckets

Error from AIStore in completing the request

CreateLocalBucket

Creates an ais bucket with a given name

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the bucket

Return

Error from AIStore in completing the request

CopyBucket

Copies existing fromBck bucket to the destination toBck thus, effectively, creating a copy of the fromBck.

AIS will create toBck on the fly, but only if the destination bucket does not exist and is provided by AIStore (note that non-AIS toBck must exist for the copy operation to be successful)
There are no limitations on copying buckets across Backend providers: you can copy AIS bucket to (or from) AWS bucket, and the latter to Google or Azure bucket, etc.
Copying multiple buckets to the same destination bucket is also permitted.

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
fromBck	string	Name of the existing bucket
toBck	string	Name of the destination bucket

RenameLocalBucket

Rename an existing bucket to the new name provided

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
oldName	string	Name of the existing bucket
newName	string	New name for the existing bucket

Return

Error from AIStore in completing the request

DestroyLocalBucket

Removes an ais bucket using its name as the identifier

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the existing bucket

Return

Error from AIStore in completing the request

EvictRemoteBucket

Evicts a remote bucket using its name as the identifier

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the existing bucket
query	url.Values	Optional URL query values such as `?provider`

Return

Error from AIStore in completing the request

SetBucketProps

Sets the properties of a bucket via action message, using the bucket name as the identifier and the bucket properties to be set

Need to set all bucket properties

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the existing bucket
props	cmn.BucketProps	Bucket properties to be set
query	url.Values	Optional URL query values such as `?provider`

Return

Error from AIStore in completing the request

ResetBucketProps

Resets the properties of a bucket, identified by its name, to the global configuration

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the existing bucket
query	url.Values	Optional URL query values such as `?provider`

Return

Error from AIStore in completing the request

Object

HeadObject

Returns the size and version of an object identified by a combination of its bucket and object names

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the bucket storing the object
provider	string	"ais" (default, can be omitted) or one of the supported backend providers: "ais", "gcp", "aws", "azure", "http", "hdfs".
object	string	Name of the object

Return

cmn.ObjectProps structure with object size, version, checksum, access time, number of copies, and other information about the object

Error from AIStore in completing the request

GetObject

Returns the size of the object. Does not validate checksum of the object in the response

Writes the response body to a writer if one is specified in the optional GetObjectInput.Writer Otherwise, it discards the response body read.

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the bucket storing the object
object	string	Name of the object
options	GetObjectInput	Optional field with a custom Writer and URL Query values

Return

Size of the object computed from the number of bytes read

Error from AIStore in completing the request

GetObjectWithValidation

Same behavior as GetObject, but performs checksum validation of the object by comparing the checksum in the response header with the calculated checksum value derived from the returned object.

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the bucket storing the object
object	string	Name of the object
options	GetObjectInput	Optional field with a custom Writer and URL Query values

Return

Size of the object computed from the number of bytes read

Error from AIStore in completing the request

PutObject

Creates an object from the body of the cmn.ReadOpenCloser argument and puts it in the bucket identified by its name. The name of the object put is likewise identified by its name. If the object hash passed in is not empty, the value is set in the request header with the default checksum type "xxhash"

Parameters

Name	Type	Description
args	PutObjectArgs	A field that handles the arguments for PutObject

PutObjectArgs

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
Bucket	string	Name of the bucket storing the object
Provider	string	"ais" (default, can be omitted) or one of the supported backend providers: "ais", "gcp", "aws", "azure", "hdfs".
Object	string	Name of the object
Hash	string	Hash computed for the object
Reader	cmn.ReadOpenCloser	Interface used to read the bytes of object data

Return

Error from AIStore in completing the request

RenameObject

Renames an existing object

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the bucket storing the object
oldName	string	Name of the existing object
newName	string	New name for the existing object

Return

Error from AIStore in completing the request

ReplicateObject

Replicates a given object

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the bucket storing the object
object	string	Name of the object to be replicated

Return

Error from AIStore in completing the request

DeleteObject

Deletes an object identified by the combination of its bucket and object name

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the bucket storing the object
object	string	Name of the object to be replicated
provider	string	"ais" (default, can be omitted) or one of the supported backend providers: "ais", "gcp", "aws", "azure", "hdfs".

Return

Error from AIStore in completing the request

EvictObject

Evicts an object identified by the combination of its bucket and object name

Parameters

Name	Type	Description
httpClient	*http.Client	HTTP Client used to create and process the HTTP Request and return the HTTP Response
proxyURL	string	URL of the proxy (gateway)
bucket	string	Name of the bucket storing the object
object	string	Name of the object to be evicted

Return

Error from AIStore in completing the request

Optional Parameters

ParamsOptional

Name	Type	Description
Query	url.Values	Map of string keys to slice of strings values, used to set the URL.RawQuery field of the HTTP Request
Headers	http.Header	Map of string keys to string values, used to set the HTTP Request header

URL Query Values

Name	Fields	Description
provider	Backend provider	"ais" (default, can be omitted) or one of the supported backend providers: "ais", "gcp", "aws", "azure", "http", "hdfs".

Basic API Workflow

A sample demo of the APIs listed above:

func demo() error {
	var (
        httpClient = &http.Client{}
        url = "http://localhost:8080"
        bucket = "DemoBucket"
        object = "DemoObject"
	)

    // Fetch cluster map
    smap, err := api.GetClusterMap(httpClient, url)
    if err != nil {
        return Errors.New("Getting clustermap failed, %v\n", err)
    }

    primaryProxyURL := smap.ProxySI.PublicNet.DirectURL

    // Create ais bucket
    err = api.CreateLocalBucket(httpClient, primaryProxyURL, bucket)
    if err != nil {
        return Errors.New("Creating ais bucket failed, %v\n", err)
    }

    newBucketName = "DemoBucketNew"
    // Rename ais bucket
    err = api.RenameLocalBucket(httpClient, primaryProxyURL, bucket, newBucketName)
    if err != nil {
        return Errors.New("Renaming ais bucket failed, %v\n", err)
    }
}

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

AIStore Client API

Overview

Types of Operations

Cluster

GetClusterMap

Parameters

Return

SetPrimaryProxy

Parameters

Return

SetClusterConfig

Parameters

Return

RegisterNode

Parameters

Return

UnregisterNode

Parameters

Return

Daemon

GetMountpaths

Parameters

Return

AddMountpath

Parameters

Return

RemoveMountpath

Parameters

Return

EnableMountpath

Parameters

Return

DisableMountpath

Parameters

Return

GetDaemonConfig

Parameters

Return

SetDaemonConfig

Parameters

Return

Bucket

HeadBucket

Parameters

Return

ListBuckets

Parameters

Return

CreateLocalBucket

Parameters

Return

CopyBucket

Parameters

RenameLocalBucket

Parameters

Return

DestroyLocalBucket

Parameters

Return

EvictRemoteBucket

Parameters

Return

SetBucketProps

Parameters

Return

ResetBucketProps

Parameters

Return

Object

HeadObject

Parameters

Return

GetObject

Parameters