diff --git a/openstack-object-storage-dev/samples/object-get-range-req.txt b/openstack-object-storage-dev/samples/object-get-range-req.txt new file mode 100644 index 0000000..0e5a782 --- /dev/null +++ b/openstack-object-storage-dev/samples/object-get-range-req.txt @@ -0,0 +1,4 @@ +GET //// HTTP/1.1 +Host: storage.swiftdrive.com +X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb +Range: bytes=4-6 diff --git a/openstack-object-storage-dev/samples/object-get-range-resp.txt b/openstack-object-storage-dev/samples/object-get-range-resp.txt new file mode 100644 index 0000000..69b601b --- /dev/null +++ b/openstack-object-storage-dev/samples/object-get-range-resp.txt @@ -0,0 +1,10 @@ +HTTP/1.1 206 Partial Content +Content-Length: 3 +Content-Type: application/octet-stream +Accept-Ranges: bytes +Last-Modified: Wed, 11 Dec 2013 01:03:26 GMT +Content-Range: bytes 4-6/10 +Etag: 3749f52bb326ae96782b42dc0a97b4c1 +Date: Wed, 11 Dec 2013 06:13:04 GMT + +456 diff --git a/openstack-object-storage-dev/samples/object-get-ranges-req.txt b/openstack-object-storage-dev/samples/object-get-ranges-req.txt new file mode 100644 index 0000000..fbce160 --- /dev/null +++ b/openstack-object-storage-dev/samples/object-get-ranges-req.txt @@ -0,0 +1,4 @@ +GET //// HTTP/1.1 +Host: storage.swiftdrive.com +X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb +Range: bytes=1-3,2-5 diff --git a/openstack-object-storage-dev/samples/object-get-ranges-resp.txt b/openstack-object-storage-dev/samples/object-get-ranges-resp.txt new file mode 100644 index 0000000..f399e13 --- /dev/null +++ b/openstack-object-storage-dev/samples/object-get-ranges-resp.txt @@ -0,0 +1,20 @@ +HTTP/1.1 206 Partial Content +Content-Length: 265 +Content-Type: multipart/byteranges;boundary=4789b20f24cc4d2a8da2e552e151e6fe +Accept-Ranges: bytes +Last-Modified: Wed, 11 Dec 2013 01:03:26 GMT +Etag: 3749f52bb326ae96782b42dc0a97b4c1 +Date: Wed, 11 Dec 2013 06:13:41 GMT + + +--4789b20f24cc4d2a8da2e552e151e6fe +Content-Type: application/octet-stream +Content-Range: bytes 1-3/10 + +123 +--4789b20f24cc4d2a8da2e552e151e6fe +Content-Type: application/octet-stream +Content-Range: bytes 2-5/10 + +2345 +--4789b20f24cc4d2a8da2e552e151e6fe-- diff --git a/openstack-object-storage-dev/section_object-api-storage-object-svc.xml b/openstack-object-storage-dev/section_object-api-storage-object-svc.xml index 1ae65f0..7b477e7 100644 --- a/openstack-object-storage-dev/section_object-api-storage-object-svc.xml +++ b/openstack-object-storage-dev/section_object-api-storage-object-svc.xml @@ -11,38 +11,23 @@ PUT'> POST'> DELETE'> - - - - - -'> - - - - - -'> ]>
Storage Object Services - An object represents the data and any metadata for the files - stored in the system. Through the ReST interface, metadata for - an object can be included by adding custom HTTP headers to the - request and the data payload as the request body. Objects - cannot exceed 5GB and must have names that do not exceed 1024 - bytes after URL encoding. However, objects larger than 5GB can - be segmented and then concatenated together so that you can - upload 5 GB segments and download a single concatenated - object. You can work with the segments and manifests directly - with HTTP requests. + An object represents the data and metadata for the files + stored in the system. Through the ReST interface, you can + include metadata for an object by adding custom HTTP headers + to the request and the data payload as the request body. + Objects cannot exceed 5 GB and must have names that do not + exceed 1024 bytes after URL encoding. However, you can segment + a large object into 5 GB segments and upload the segments. + Then, you can download the segments as a single concatenated + object. You can use HTTP requests to work with the segments + and manifests directly. - Verb @@ -91,84 +76,151 @@ format="SVG" scale="60"/>
Get Object Details - &GET; operations against an object are used to retrieve - the object's data. - Note that you can perform conditional &GET; requests by - using certain HTTP headers as documented in RFC 2616. - OpenStack Object Storage supports the following - headers: - RFC 2616: Perform &GET; operations against an object to get object + data. + You can perform conditional &GET; requests by using the + following HTTP headers in the request: + + + If-Match + + + If-None-Match + + + If-Modified-Since + + + If-Unmodified-Since + + + These headers are documented in http://www.ietf.org/rfc/rfc2616.txt - + >http://www.ietf.org/rfc/rfc2616.txt. + You can use the HTTP Range header to fetch + portions of data by using one or more range + specifications. To specify many ranges, separate the range + specifications with a comma. + The types of range specifications are: - If-Match + Byte range + specification. Use + FIRST_BYTE_OFFSET to specify the start of the data + range, and LAST_BYTE_OFFSET to specify the end. + You can omit the LAST_BYTE_OFFSET and if you do, + the value defaults to the offset of the last byte + of data. - If-None-Match - - - If-Modified-Since - - - If-Unmodified-Since - - - It is also possible to fetch a portion of data using the - HTTP Range header. At this time, OpenStack - Object Storage does not support the full specification for - Range but basic support is provided. - OpenStack Object Storage only allows a single range that - includes OFFSET and/or LENGTH. We support a sub-set of - Range and do not adhere to the full - RFC-2616 specification. We support specifying - OFFSET-LENGTH where either OFFSET or LENGTH can be - optional (not both at the same time). The following are - supported forms of the header: - - - Range: bytes=-5 - last five bytes - of the object - - - Range: bytes=10-15 - the five bytes - after a 10-byte offset - - - Range: bytes=32- - all data after - the first 32 bytes of the object + Suffix byte range + specification. Use LENGTH bytes to + specify the length of the data range. + The following forms of the header specify the following + ranges of data: + + + + + + Header + Range of object data + + + + + + Range: bytes=-5 + The last 5 bytes. + + + + + Range: + bytes=4-6 + Bytes 4 to 6 inclusive. + + + + Range: + bytes=2-2 + Byte 2, the third byte of the + data. + + + + + Range: bytes=6- + Byte 6 byte and after. + + + + + Range: + bytes=1-3,2-5 + Bytes 1 to 3 inclusive, and bytes 2 to 5 + inclusive. + The Content-Type of the + response is then + multipart/byteranges. + + + + Get Object Details HTTP Request - The object's data is returned in the response body. - Object metadata is returned as HTTP headers. A status of 2xx - (between 200 and 299, inclusive) indicates success; status 404 - (Not Found) is returned if no such object exists. + The object data is returned in the response body. Object + metadata is returned as HTTP headers. A status of + 2xx (between 200 and 299, + inclusive) indicates success; status 404 (Not Found) is + returned if no such object exists. Get Object Details HTTP Response - [ ... ] + + These examples include an object that contains 10 bytes + of data: 0123456789. + + Get Object Details HTTP Request Using Range + + + + Get Object Details HTTP Response When Using + Range + + + + Get Object Details HTTP Request Using Multiple + Ranges + + + + Get Object Details HTTP Response When Using + Multiple Ranges +
Create or Update Object &PUT; operations are used to write, or overwrite, an - object's content and metadata. + object's content and metadata. You can ensure end-to-end data integrity by including an MD5 checksum of your object's data in the ETag header. You are not required to include the ETag header, but it is recommended to ensure that the storage system successfully stored your object's content. - You can cause an object to expire after a certain date by using - the X-Delete-At or X-Delete-After headers - during an object &PUT; operation. When Object Storage detects one of - these headers, the system automatically stops serving that object at - the specified time and shortly after the expiration date, it removes - the object from the storage system. + You can cause an object to expire after a certain date + by using the X-Delete-At or + X-Delete-After headers during an object + &PUT; operation. When Object Storage detects one of these + headers, the system automatically stops serving that + object at the specified time and shortly after the + expiration date, it removes the object from the storage + system. The HTTP response will include the MD5 checksum of the data written to the storage system. If you do not send the ETag in the request, you should compare the value returned @@ -183,8 +235,8 @@ format="SVG" scale="60"/> The object can be created with custom metadata via HTTP headers identified with the X-Object-Meta- prefix, or arbitrary headers set with the - allowed_headers option in the object-server - configuration. + allowed_headers option in the + object-server configuration. Create or Update Object HTTP Request @@ -219,7 +271,7 @@ format="SVG" scale="60"/> transfer will be less than 5GB or for splitting it into 5GB chunks, each in its own storage object. If you have files that are larger than 5GB see . + linkend="large-object-creation"/>. Upload Unspecified Quantity of Content HTTP Request @@ -263,32 +315,33 @@ format="SVG" scale="60"/> If you use the © operation using a manifest object as the source, the new object is a "normal" object (not segmented). If the total size of the - source segment objects exceeds 5GB, the © - operation will fail. However, as explained later, you + source segment objects exceeds 5 GB, the © + operation fails. However, as explained later, you can make a duplicate of the manifest object. This new - object may be larger than 5GB. + object can be larger than 5 GB. - There are two types of manifest object as follows: - - Static Large Objects. The manifest object - content is an ordered list of the names of the - segment objects in json format. - - - Dynamic Large Objects. The manifest object - has no content. However, it has - X-Object-Manifest metadata. - The value of this is - <container>/<prefix>, - where <container> is the name - of the container where the segment objects are - stored and <prefix> is a - string that all the segment objects have in - common. - - While both types of manifest objects have - similar behavior, there are differences as explained in - the following table. + The manifest object type are: + + + Static large objects. The manifest object + content is an ordered list of the names of the + segment objects in json format. + + + Dynamic large objects. The manifest object has + no content. However, it has + X-Object-Manifest metadata. The + value of this is + <container>/<prefix>, where + <container> is the name of the + container where the segment objects are stored and + <prefix> is a string that all + the segment objects have in common. + + + While both types of manifest objects have similar + behavior, there are differences as explained in the + following table. @@ -402,14 +455,19 @@ format="SVG" scale="60"/> manifest object streams all the segments concatenated. There is no limit to the number of segments that can be a part of a single large object. - To ensure the download works correctly, you must upload all the object segments to - the same container and ensure that each object name is prefixed in such a way that - it sorts in the order in which it should be concatenated. You also create and upload - a manifest file. The manifest file is a zero-byte file with the extra - X-Object-Manifest: <container>/<prefix> header, where <container> - is the container the object segments are in and <prefix> is the common prefix - for all the segments. The container and common prefix must be UTF-8 encoded and - URL-encoded in the X-Object-Manifest header. + To ensure the download works correctly, you must + upload all the object segments to the same container + and ensure that each object name is prefixed in such a + way that it sorts in the order in which it should be + concatenated. You also create and upload a manifest + file. The manifest file is a zero-byte file with the + extra X-Object-Manifest: + <container>/<prefix> header, where + <container> is the container the object segments + are in and <prefix> is the common prefix for all + the segments. The container and common prefix must be + UTF-8 encoded and URL-encoded in the X-Object-Manifest + header.It is best to upload all the segments first and then create or update the manifest. With this method, the full object will not be available for downloading @@ -431,9 +489,8 @@ format="SVG" scale="60"/> No response body is returned. A status code of 2xx (between 200 and 299, inclusive) indicates a - successful write; status 411 (Length Required) - denotes a missing - Content-Length or + successful write; status 411 (Length Required) denotes + a missing Content-Length or Content-Type header in the request. If the MD5 checksum of the data written to the storage system does NOT match the (optionally) supplied ETag @@ -474,60 +531,59 @@ format="SVG" scale="60"/>
Static Large Objects - A Static Large Object is created in two steps: - - Divide your content into pieces and - create (i.e., upload) a segment object to - contain each piece. You must record the - ETag response header - returned by the&PUT; operation. - Alternatively, you can calculate the MD5 - checksum of the segment prior to uploading - and include this in the ETag - request header -- this ensures that the - upload cannot corrupt your data. - - - List the name of each segment object - along with its size and MD5 checksum in - order. Create a manifest object. You - indicate that this is a manifest object by - including the - ?multipart-manifest=put - query string at the end of the manifest - object name. - - The body of the &PUT; request on the - manifest object comprises a json list, where each - element contains the following: - - path - this is the - container and object name in the following - format: - <container-name>/<object-name> - - - etag - this is the MD5 - checksum of the content of the segment - object. This must match the - ETag of that - object. - - - size_bytes - this is the - size of the segment object. This must - match the Content-Length of - that object - - - + To create a static large object: + + + Divide your content into pieces and create + (upload) a segment object to contain each + piece. You must record the ETag + response header returned by the&PUT; + operation. Alternatively, you can calculate + the MD5 checksum of the segment prior to + uploading and include this in the + ETag request header. This + ensures that the upload cannot corrupt your + data. + + + List the name of each segment object along + with its size and MD5 checksum in order. + Create a manifest object. You indicate that + this is a manifest object by including the + ?multipart-manifest=put query + string at the end of the manifest object + name. + + + The body of the &PUT; request on the manifest object + comprises a json list, where each element contains the + following: + + + path - this is the container + and object name in the following format: + <container-name>/<object-name> + + + etag - this is the MD5 checksum + of the content of the segment object. This + must match the ETag of that + object. + + + size_bytes - this is the size + of the segment object. This must match the + Content-Length of that + object + + Static Large Object Manifest List This is an example containing three segment - objects. In this example, we illustrate that in - contrast to dynamic large objects, you can use a - number of containers and the object names do not - have to conform to a specific pattern. + objects. In this example, you can use several + containers and the object names do not have to + conform to a specific pattern, in contrast to + dynamic large objects. The Content-Length request header must @@ -561,7 +617,7 @@ format="SVG" scale="60"/> ?multipart-manifest=get. The resulting list will not be identically formatted as the manifest you originally used in the &PUT; - operation. + operation. If you use the &DELETE; operation on a manifest object, the manifest object is deleted -- the segment objects are not affected. However, if you add the @@ -579,11 +635,12 @@ format="SVG" scale="60"/>
Assigning CORS Headers to Requests - CORS is a specification that stands for Cross-Origin Resource - Sharing. It defines how browsers and servers communicate across - origins using HTTP headers, such as those assigned by Object Storage - API requests. These headers are supported with the Object Storage - API. You can read more about the definition of the Access-Control- + CORS is a specification that stands for Cross-Origin + Resource Sharing. It defines how browsers and servers + communicate across origins using HTTP headers, such as + those assigned by Object Storage API requests. These + headers are supported with the Object Storage API. You can + read more about the definition of the Access-Control- response headers and Origin response header at www.w3.org/TR/access-control/. @@ -612,13 +669,14 @@ format="SVG" scale="60"/> Origin - You can assign these headers to objects only. + You can assign these headers to objects only. Assign CORS Header HTTP Request - In the example, the origin header is assigned that indicates - where the file came from. This allows you to provide security - that requests to your Object Storage repository are indeed from - the correct origination: + In the example, the origin header is assigned that + indicates where the file came from. This allows you to + provide security that requests to your Object Storage + repository are indeed from the correct + origination:
@@ -670,7 +728,7 @@ format="SVG" scale="60"/> By setting the header to a specific Epoch time, you indicate when you want the object to expire, not be served, and be - deleted completely from the storage system. + deleted completely from the storage system.The X-Delete-After header takes an integer number of seconds and calculates the amount of time from now that you want the object to be deleted. The proxy @@ -704,7 +762,7 @@ format="SVG" scale="60"/>
Object Versioning - Object Versioning allows you to store multiple versions + Object versioning allows you to store multiple versions of your content to recover from unintended overwrites. It provides an easy method to implement version control which can be used on any type of content. It is strongly @@ -715,23 +773,28 @@ format="SVG" scale="60"/> in that container copy the prior object to a separate "non-current version" container. Each of the non-current versions of an object has a time stamp appended to it, so - you know when it was created. - To enable Object Versioning, your cloud provider has to set - allow_versions to TRUE in their container config. Then, - create a container where your non-current versions will be written. Next, set the - metadata X-Versions-Location header on the container that holds the current - versions of your objects. Set the metadata header to point to the new non-current - version container you created. The name of the container must be UTF-8 encoded and then - URL-encoded before putting into the X-Versions-Location header. This is - where your non-current versions will be stored. Once this is done, each object in your - current-version container will have Object Versioning enabled; changes to the objects - automatically create non-current versions in the separate container. + you know when it was created. + To enable object versioning, your cloud provider has to + set allow_versions to TRUE in + their container config. Then, create a container where + your non-current versions will be written. Next, set the + metadata X-Versions-Location header on the + container that holds the current versions of your objects. + Set the metadata header to point to the new non-current + version container you created. The name of the container + must be UTF-8 encoded and then URL-encoded before putting + into the X-Versions-Location header. This is + where your non-current versions will be stored. Once this + is done, each object in your current-version container + will have Object Versioning enabled; changes to the + objects automatically create non-current versions in the + separate container. Nothing is written to the non-current version container when you initially &PUT; an object into the current-version container. Only when you make edits to the objects via &PUT; will you create non-current versions. These non-current versions are labeled according to the - schema below. + schema below. Naming Schema: Non-current versions are assigned the name @@ -739,37 +802,37 @@ format="SVG" scale="60"/> length is the 3-character zero-padded hexadecimal character length of the <object_name> and <timestamp> is when the it was initially created as - a current version. + a current version. Any return status in the 2xx range, such as 202 - (Accepted), denotes success. Status codes in the 4xx or 5xx - range denote failure. You should retry your request if you - receive an error. Please note, however, that if you have - specified a container that does not exist as your + (Accepted), denotes success. Status codes in the 4xx or + 5xx range denote failure. You should retry your request if + you receive an error. Please note, however, that if you + have specified a container that does not exist as your non-current version container, a status of 412 (Precondition Failed) returns when you edit the versioned object. If you receive this error, check that the - container exists. + container exists. A &GET; to a versioned object returns the current version of the object without having to do any request - redirects or metadata lookups. + redirects or metadata lookups. A &POST; to a versioned object only updates the object's metadata; it does not create a new version of the object. In other words, new versions are only created when the - content of the object changes. + content of the object changes. A &DELETE; to a versioned object removes the current version of the object and replaces it with the next-most current version, moving it from the non-current container to the current. This next-most current version carries with it any metadata last set on it. If want to completely remove an object and you have five total versions of it, - you must &DELETE; it five times. + you must &DELETE; it five times. A large-object manifest file cannot be versioned, - but it may point to versioned segments. + but it may point to versioned segments. To turn off Object Versioning on your current version container, remove its X-Versions-Location - metadata by sending an empty key value. + metadata by sending an empty key value. Object Versioning with cURL Make sure a version-storing container exists, @@ -780,39 +843,30 @@ format="SVG" scale="60"/> also add the X-Versions-Location header to an existing container. In this example, the name of the container is “versions”; the location for the - current version is the container "current". + current version is the container "current". Create a container named versions. - curl -i -XPUT -H "X-Auth-Token: <token>" http://<storage_url>/versions + $ curl -i -XPUT -H "X-Auth-Token: <token>" http://<storage_url>/versions Create a container named current with the X-Versions-Location header that references "versions". - - curl -i -XPUT -H "X-Auth-Token: <token>" \ - -H "X-Versions-Location: versions" http://<storage_url>/current - - Create an object (the first version): - - curl -i -XPUT --data-binary 1 -H "X-Auth-Token: <token>" \ - http://<storage_url>/current/myobject - - Now create a new version of that object: - - curl -i -XPUT --data-binary 2 -H "X-Auth-Token: <token>" \ - http://<storage_url>/current/myobject - - See a listing of the older versions of the object: - - curl -i -H "X-Auth-Token: <token>" \ - http://<storage_url>/versions?prefix=008myobject/ - + $ curl -i -XPUT -H "X-Auth-Token: <token>" \ + -H "X-Versions-Location: versions" http://<storage_url>/current + Create an object (the first version): + $ curl -i -XPUT --data-binary 1 -H "X-Auth-Token: <token>" \ + http://<storage_url>/current/myobject + Now create a new version of that object: + $ curl -i -XPUT --data-binary 2 -H "X-Auth-Token: <token>" \ + http://<storage_url>/current/myobject + See a listing of the older versions of the + object: + $ curl -i -H "X-Auth-Token: <token>" \ + http://<storage_url>/versions?prefix=008myobject/ Now delete the current version of the object and see - that the older version is gone: - - curl -i -XDELETE -H "X-Auth-Token: <token>" \ - http://<storage_url>/current/myobject - curl -i -H "X-Auth-Token: <token>" \ - http://<storage_url>/versions?prefix=008myobject/ - + that the older version is gone: + $ curl -i -XDELETE -H "X-Auth-Token: <token>" \ + http://<storage_url>/current/myobject + $ curl -i -H "X-Auth-Token: <token>" \ + http://<storage_url>/versions?prefix=008myobject/
@@ -825,13 +879,16 @@ format="SVG" scale="60"/> you can save the step of re-uploading the content and thus also save the associated bandwidth charges, if any were to apply. - There are two ways to copy an existing object to another object in OpenStack Object - Storage. One way is to do a &PUT; to the new object (the target) location, but add the - “X-Copy-From” header to designate the source of the data. The header - value should be the container and object name of the source object in the form of - “/container/object”. The container and object name must be UTF-8 encoded and then - URL-encoded. Also, the X-Copy-From &PUT; requests require a Content-Length - header, even if it is zero (0). + There are two ways to copy an existing object to another + object in OpenStack Object Storage. One way is to do a + &PUT; to the new object (the target) location, but add the + “X-Copy-From” header to designate the + source of the data. The header value should be the + container and object name of the source object in the form + of “/container/object”. The container and object name must + be UTF-8 encoded and then URL-encoded. Also, the + X-Copy-From &PUT; requests require a + Content-Length header, even if it is zero (0). Object Copy Method 1 @@ -840,7 +897,7 @@ format="SVG" scale="60"/> © to the existing object, and include the “Destination” header to specify the target of the copy. The header value is the container and new object name in - the form of “/container/object”. + the form of “/container/object”. Object Copy Method 2 @@ -875,7 +932,7 @@ format="SVG" scale="60"/> conflicting keys on the target (new) object. One interesting use case is to copy an object to itself and set the content type to a new value. This is the only way - to change the content type of an existing object. + to change the content type of an existing object.
Delete Object @@ -896,14 +953,14 @@ format="SVG" scale="60"/> deleted within one day of the expiration time and the object is not served immediately after the expiration time. Refer to Expiring Objects for more details. + >Expiring Objects for more details. Object Delete HTTP Request No response body is returned. A status code of 2xx ( - between 200 and 299, inclusive) indicates success; status code - 404 (Not Found) is returned when the object does not + between 200 and 299, inclusive) indicates success; status + code 404 (Not Found) is returned when the object does not exist. Object Delete HTTP Response @@ -923,7 +980,7 @@ format="SVG" scale="60"/> No response body is returned. Metadata is returned as HTTP headers. A status code of 2xx (between 200 and 299, inclusive) indicates success; status 404 (Not Found) is - returned when the object does not exist. + returned when the object does not exist. Get Object Metadata HTTP Response @@ -942,7 +999,7 @@ format="SVG" scale="60"/> storage objects (see &PUT;). Also refer to copying an object when you need to update metadata or other headers such as - Content-Type or CORS headers. + Content-Type or CORS headers. Key names must be prefixed with X-Object-Meta-. A &POST; request will delete all existing metadata added with a previous @@ -952,9 +1009,9 @@ format="SVG" scale="60"/> No response body is returned. A status code of 2xx - (between 200 and 299, inclusive) indicates success; status 404 - (Not Found) is returned if the requested object does not - exist. + (between 200 and 299, inclusive) indicates success; status + 404 (Not Found) is returned if the requested object does + not exist. Update Object Metadata HTTP Response
Comparison of Static and Dynamic Large Objects