The default AWS rate limit is 2 instances/sec, but in practice, we
can achieve something like 0.6 instances/sec with the current code.
That's because the create instance REST API call itself takes more
than a second to return. To achieve even the default AWS rate
(much less a potentially faster one which may be obtainable via
support request), we need to alter the approach. This change does
the following:
* Paralellizes create API calls. We create a threadpool with
(typically) 8 workers to execute create instance calls in the
background. 2 or 3 workers should be sufficient to meet the
2/sec rate, more allows for the occasional longer execution time
as well as a customized higher rate. We max out at 8 to protect
nodepool from too many threads.
* The state machine uses the new background create calls instead
of synchronously creating instances. This allows other state
machines to progress further (ie, advance to ssh keyscan faster
in the case of a rush of requests).
* Delete calls are batched. They don't take as long as create calls,
yet their existence at all uses up rate limiting slots which could
be used for creating instances. By batching deletes, we make
more room for creates.
* A bug in the RateLimiter could cause it not to record the initial
time and therefore avoid actually rate limiting. This is fixed.
* The RateLimiter is now thread-safe.
* The default rate limit for AWS is changed to 2 requests/sec.
* Documentation for the 'rate' parameter for the AWS driver is added.
* Documentation for the 'rate' parameter for the Azure driver is
corrected to describe the rate as requests/sec instead of delay
between requests.
Change-Id: Ida2cbc59928e183eb7da275ff26d152eae784cfe
d5b0dee642