ed4f3c4db9
Point to WADL files that generate the API Reference
page so that the method descriptions and request
and response parameters are consistent between
these two docs:
http://api.openstack.org/api-ref-objectstorage.html
(API Reference page for Object Storage)
http://docs.openstack.org/api/openstack-object-storage/1.0/
(API Reference (or spec) for Object Storage)
Add descriptions of ACLs, FormPOST, TempURLs,
StaticWeb, Bulk Upload, Bulk Delete, the OPTIONS operation
(though this is implicit in CORS support)
Validate all code examples
Remove duplication.
Co-Author: Donagh McCabe
Closes-Bug: #1187119
Closes-Bug: #1214139
Closes-Bug: #1074198
Partial-Bug: #1255770
Change-Id: I94054b046a94260ba8825bdb42439adfcaf9fdce
author: diane fleming
370 lines
21 KiB
XML
370 lines
21 KiB
XML
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE section [
|
|
<!-- Some useful entities borrowed from HTML -->
|
|
<!ENTITY ndash "–">
|
|
<!ENTITY mdash "—">
|
|
<!ENTITY hellip "…">
|
|
<!ENTITY nbsp " ">
|
|
<!-- Useful for describing APIs -->
|
|
<!ENTITY COPY '<command xmlns="http://docbook.org/ns/docbook">COPY</command>'>
|
|
<!ENTITY GET '<command xmlns="http://docbook.org/ns/docbook">GET</command>'>
|
|
<!ENTITY HEAD '<command xmlns="http://docbook.org/ns/docbook">HEAD</command>'>
|
|
<!ENTITY PUT '<command xmlns="http://docbook.org/ns/docbook">PUT</command>'>
|
|
<!ENTITY POST '<command xmlns="http://docbook.org/ns/docbook">POST</command>'>
|
|
<!ENTITY DELETE '<command xmlns="http://docbook.org/ns/docbook">DELETE</command>'>
|
|
]>
|
|
<section xmlns="http://docbook.org/ns/docbook"
|
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
|
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0"
|
|
xml:id="large-object-creation">
|
|
<title>Large objects</title>
|
|
<para>By default, the content of an object cannot be greater than
|
|
5 GB. However, you can use segment objects and manifest
|
|
objects to store more content.</para>
|
|
<section xml:id="segment-objects">
|
|
<title>Segment objects</title>
|
|
<para>You can divide your content into segments, and upload
|
|
each segment into its own segment object. Segment objects
|
|
do not have any special features. You create, update,
|
|
download, and delete segment objects just as you would
|
|
normal objects.</para>
|
|
</section>
|
|
<section xml:id="manifest-objects">
|
|
<title>Manifest objects</title>
|
|
<para>A manifest object points to segment objects. When you
|
|
download a manifest object, Object Storage concatenates
|
|
the contents of the segment objects and returns this in
|
|
the response body of the request.</para>
|
|
<para>This behavior extends to the response headers returned
|
|
by &GET; and &HEAD; requests. The
|
|
<literal>Content-Length</literal> response header
|
|
value is the total size of all segment objects. Object
|
|
Storage calculates the <literal>ETag</literal> response
|
|
header value by taking the <literal>ETag</literal> value
|
|
of each segment, concatenating them together, and
|
|
returning the MD5 checksum of the result.</para>
|
|
<note>
|
|
<para>If you make a © request by using a manifest
|
|
object as the source, the new object is a normal, and
|
|
not a segment, object. If the total size of the source
|
|
segment objects exceeds 5 GB, the © request
|
|
fails. However, you can make a duplicate of the
|
|
manifest object and this new object can be larger than
|
|
5 GB.</para>
|
|
</note>
|
|
<para>The manifest object types are:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><emphasis role="bold">Static large
|
|
objects</emphasis>. The manifest object
|
|
content is an ordered list of the names of the
|
|
segment objects in JSON format.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><emphasis role="bold">Dynamic large
|
|
objects</emphasis>. The manifest object has no
|
|
content.</para>
|
|
<para>However, it has
|
|
<literal>X-Object-Manifest</literal> metadata
|
|
header. The value of this header is
|
|
<literal><container>/<prefix></literal>,
|
|
where <literal><container></literal> is the
|
|
name of the container where the segment objects
|
|
are stored, and <literal><prefix></literal> is
|
|
a string that all segment objects have in
|
|
common.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>While both types of manifest objects have similar
|
|
behavior, the following table describes their
|
|
differences:</para>
|
|
<table rules="all">
|
|
<caption>Static and dynamic large objects</caption>
|
|
<thead>
|
|
<tr>
|
|
<th>Object type</th>
|
|
<th>End-to-end integrity</th>
|
|
<th>Upload order</th>
|
|
<th>Removal or addition of segment objects</th>
|
|
<th>Segment object size and number</th>
|
|
<th>Segment object container name</th>
|
|
<th>Manifest Object Metadata</th>
|
|
<th>Making a copy of the manifest object</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td><para>Static large object</para></td>
|
|
<td><para>Assured. The list of segments includes
|
|
the MD5 checksum (<literal>ETag</literal>)
|
|
of each segment. You cannot upload the
|
|
manifest object if the
|
|
<literal>ETag</literal> in the list
|
|
differs from the segment object already
|
|
uploaded. If a segment is somehow lost, an
|
|
attempt to download the manifest object
|
|
results in an error.</para></td>
|
|
<td><para>The segment objects must be uploaded
|
|
before the manifest object.</para></td>
|
|
<td><para>You cannot add or remove segment objects
|
|
from the manifest. However, you can create
|
|
a completely new manifest object of the
|
|
same name with a different manifest
|
|
list.</para></td>
|
|
<td><para>Segment objects must be at least 1 MB in
|
|
size (by default). The final segment
|
|
object can be any size. At most 1000
|
|
segments are supported (by
|
|
default).</para></td>
|
|
<td><para>The manifest list includes the container
|
|
name of each object. Segment objects can
|
|
be in different containers.</para></td>
|
|
<td><para>The object has
|
|
<literal>X-Static-Large-Object</literal>
|
|
set to <literal>true</literal>. You do not
|
|
set this metadata directly. Instead the
|
|
system sets it when you &PUT; a static
|
|
manifest object.</para></td>
|
|
<td><para/></td>
|
|
</tr>
|
|
<tr>
|
|
<td><para>Dynamic large object</para></td>
|
|
<td><para>Not guaranteed. The eventual consistency
|
|
model means that although you have
|
|
uploaded a segment object, it might not
|
|
appear in the container listing until
|
|
later. If you download the manifest before
|
|
it appears in the container, it does not
|
|
form part of the content returned in
|
|
response to a &GET; request.</para></td>
|
|
<td><para>You can upload manifest and segment
|
|
objects in any order. You are recommended
|
|
to upload the manifest object after the
|
|
segments in case a premature download of
|
|
the manifest occurs. However, this is not
|
|
enforced.</para></td>
|
|
<td><para>You can upload new segment objects or
|
|
remove existing segments. The names must
|
|
simply match the
|
|
<literal><prefix></literal>
|
|
supplied in
|
|
<literal>X-Object-Manifest</literal>.</para></td>
|
|
<td><para>Segment objects can be of any
|
|
size.</para></td>
|
|
<td><para>All segment objects must be in the same
|
|
container</para></td>
|
|
<td><para>The <literal>X-Object-Manifest</literal>
|
|
value is the
|
|
<literal><container>/<prefix></literal>
|
|
indicating where the segment objects are
|
|
located. You supply this request header in
|
|
the &PUT; operation</para></td>
|
|
<td><para>The © operation does not create a
|
|
manifest object. To duplicate a manifest
|
|
object, use the &GET; operation to read
|
|
the value of
|
|
<literal>X-Object-Manifest</literal>
|
|
and use this value in the
|
|
<literal>X-Object-Manifest</literal>
|
|
request header in a &PUT; operation. This
|
|
creates a new manifest object that shares
|
|
the same set of segment objects as the
|
|
original manifest object.</para></td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
<section xml:id="dynamic-large-object-creation">
|
|
<title>Dynamic large objects</title>
|
|
<para>You must segment objects that are larger than 5 GB
|
|
before you can upload them. You then upload the
|
|
segment objects like you would any other object and
|
|
create a dynamic large manifest object. The manifest
|
|
object tells Object Storage how to find the segment
|
|
objects that comprise the large object. The segments
|
|
remain individually addressable, but retrieving the
|
|
manifest object streams all the segments concatenated.
|
|
There is no limit to the number of segments that can
|
|
be a part of a single large object.</para>
|
|
<para>To ensure the download works correctly, you must
|
|
upload all the object segments to the same container
|
|
and ensure that each object name is prefixed in such a
|
|
way that it sorts in the order in which it should be
|
|
concatenated. You also create and upload a manifest
|
|
file. The manifest file is a zero-byte file with the
|
|
extra <literal>X-Object-Manifest</literal>
|
|
<code><container>/<prefix></code> header,
|
|
where <code><container></code> is the container
|
|
the object segments are in and
|
|
<code><prefix></code> is the common prefix
|
|
for all the segments. You must UTF-8-encode and then
|
|
URL-encode the container and common prefix in the
|
|
<literal>X-Object-Manifest</literal>
|
|
header.</para>
|
|
<para>It is best to upload all the segments first and then
|
|
create or update the manifest. With this method, the
|
|
full object is not available for downloading until the
|
|
upload is complete. Also, you can upload a new set of
|
|
segments to a second location and then update the
|
|
manifest to point to this new location. During the
|
|
upload of the new segments, the original manifest is
|
|
still available to download the first set of
|
|
segments.</para>
|
|
<example>
|
|
<title>Upload segment of large object request:
|
|
HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/large-object-upload-segment-req.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<example>
|
|
<title>Upload segment of large object response:
|
|
HTTP</title>
|
|
<literallayout class="monospaced">s</literallayout>
|
|
</example>
|
|
<para>No response body is returned. A status code of
|
|
<returnvalue>2<replaceable>nn</replaceable></returnvalue>
|
|
(between 200 and 299, inclusive) indicates a
|
|
successful write; status <errorcode>411</errorcode>
|
|
<errortext>Length Required</errortext> denotes a
|
|
missing <literal>Content-Length</literal> or
|
|
<literal>Content-Type</literal> header in the request.
|
|
If the MD5 checksum of the data written to the storage
|
|
system does NOT match the (optionally) supplied ETag
|
|
value, a <errorcode>422</errorcode>
|
|
<errortext>Unprocessable Entity</errortext> response
|
|
is returned.</para>
|
|
<para>You can continue uploading segments like this
|
|
example shows, prior to uploading the manifest.</para>
|
|
<example>
|
|
<title>Upload next segment of large object request:
|
|
HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/large-object-upload-next-segment-req.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<example>
|
|
<title>Upload next segment of large object response:
|
|
HTTP</title>
|
|
<literallayout class="monospaced">w</literallayout>
|
|
</example>
|
|
<para>Next, upload the manifest you created that indicates
|
|
the container the object segments reside within. Note
|
|
that uploading additional segments after the manifest
|
|
is created causes the concatenated object to be that
|
|
much larger but you do not need to recreate the
|
|
manifest file for subsequent additional
|
|
segments.</para>
|
|
<example>
|
|
<title>Upload manifest request: HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/upload-manifest-req.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<example>
|
|
<title>Upload manifest response: HTTP</title>
|
|
<literallayout class="monospaced"><xi:include href="samples/upload-manifest-resp.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<para>The response's <literal>Content-Type</literal> for a
|
|
&GET; or &HEAD; on the manifest is the same as the
|
|
<literal>Content-Type</literal> set during the
|
|
&PUT; request that created the manifest. You can
|
|
easily change the <literal>Content-Type</literal> by
|
|
reissuing the &PUT; request.</para>
|
|
</section>
|
|
<section xml:id="static-large-objects">
|
|
<title>Static large objects</title>
|
|
<procedure>
|
|
<title>To create a static large object</title>
|
|
<step>
|
|
<para>Divide your content into pieces and create
|
|
(upload) a segment object to contain each
|
|
piece. You must record the <literal>ETag</literal>
|
|
response header returned by the&PUT;
|
|
operation. Alternatively, you can calculate
|
|
the MD5 checksum of the segment prior to
|
|
uploading and include this in the
|
|
<literal>ETag</literal> request header. This
|
|
ensures that the upload cannot corrupt your
|
|
data.</para>
|
|
</step>
|
|
<step>
|
|
<para>List the name of each segment object along
|
|
with its size and MD5 checksum in order.
|
|
Create a manifest object. You indicate that
|
|
this is a manifest object by including the
|
|
<parameter>?multipart-manifest=put</parameter> query
|
|
string at the end of the manifest object
|
|
name.</para>
|
|
</step>
|
|
</procedure>
|
|
<para>The body of the &PUT; request on the manifest object
|
|
comprises a json list, where each element contains the
|
|
following attributes:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><code>path</code>. The container and object
|
|
name in this format:
|
|
<code><container-name>/<object-name></code></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><code>etag</code>. The MD5 checksum of the
|
|
content of the segment object. This value must
|
|
match the <literal>ETag</literal> of that
|
|
object.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><code>size_bytes</code>. The size of the
|
|
segment object. This value must match the
|
|
<literal>Content-Length</literal> of that
|
|
object.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<example>
|
|
<title>Static large object manifest list</title>
|
|
<para>This example shows three segment objects. You
|
|
can use several containers and the object names do
|
|
not have to conform to a specific pattern, in
|
|
contrast to dynamic large objects.</para>
|
|
<literallayout class="monospaced"><xi:include href="samples/slo-manifest-example.txt" parse="text"/></literallayout>
|
|
</example>
|
|
<para>The <literal>Content-Length</literal> request header must
|
|
contain the length of the json content. Not the length
|
|
of the segment objects. However, after the &PUT;
|
|
operation completes, the <literal>Content-Length</literal>
|
|
metadata is set to the total length of all the object
|
|
segments. A similar situation applies to the
|
|
<literal>ETag</literal>. If used in the &PUT; operation,
|
|
it must contain the MD5 checksum of the json content.
|
|
The <literal>ETag</literal> metadata value is then set to be
|
|
the MD5 checksum of the concatenated <literal>ETag</literal>
|
|
values of the object segments. You can also set the
|
|
<literal>Content-Type</literal> request header and
|
|
custom object metadata.</para>
|
|
<para>When the &PUT; operation sees the
|
|
<parameter>?multipart-manifest=put</parameter>
|
|
query parameter, it reads the request body and verifies
|
|
that each segment object exists and that the sizes and
|
|
ETags match. If there is a mismatch, the
|
|
&PUT;operation fails.</para>
|
|
<para>If everything matches, the manifest object is
|
|
created. The <literal>X-Static-Large-Object</literal>
|
|
metadata is set to <literal>true</literal> indicating
|
|
that this is a static object manifest.</para>
|
|
<para>Normally when you perform a &GET; operation on the
|
|
manifest object, the response body contains the
|
|
concatenated content of the segment objects. To
|
|
download the manifest list, use the
|
|
<parameter>?multipart-manifest=get</parameter> query parameter.
|
|
The resulting list is not formatted the same as the
|
|
manifest you originally used in the &PUT;
|
|
operation.</para>
|
|
<para>If you use the &DELETE; operation on a manifest
|
|
object, the manifest object is deleted. The segment
|
|
objects are not affected. However, if you add the
|
|
<parameter>?multipart-manifest=delete</parameter> query parameter,
|
|
the segment objects are deleted and if all are
|
|
successfully deleted, the manifest object is also
|
|
deleted.</para>
|
|
<para>To change the manifest, use a &PUT; operation with
|
|
the <parameter>?multipart-manifest=put</parameter>
|
|
query parameter. This request creates a
|
|
new manifest object. You can
|
|
also update the object metadata in the usual
|
|
way.</para>
|
|
</section>
|
|
</section>
|
|
</section>
|