Cloud Scheduler REST API
The APIs provided by
admin_server.py
and
info_server.py
and consumed by
cloud_admin
and
cloud_status
have been transitioned from XML-RPC to REST.
SimpleXMLRPCServer
and
xmlrpclib
have been replaced with
web.py
and
requests
. This page provides information regarding this transition.
API Design
After a review of the server and client, the following URL schema was proposed. The HTTP method, URL and parameters, and replaced XML-RPC functions are listed. The API follows a RESTful approach, where URLs represent resources and HTTP methods represent verbs that apply to the resources.
GET
requests are safe,
PUT
requests are unsafe but idempotent, and
POST
and
DELETE
are neither.
cloud_admin
PUT /?log_level=LOG_LEVEL
change_log_level
POST /?action=reconfig|quick_shutdown
cloud_resources_reconfig
perform_quick_shutdown
PUT /clouds/CLOUD?action=(enable|disable)
enable_cloud
disable_cloud
PUT /clouds/CLOUD?allocations=NUMBER
adjust_cloud_allocation
DELETE /clouds/CLOUD/vms
delete_all_vm_entry_cloud
PUT /clouds/CLOUD/vms?action=shutdown[&count=all|NUMBER]
shutdown_cluster_count
shutdown_cluster_all
POST /clouds/CLOUD/vms?action=force_retire[&count=all|NUMBER]
force_retire_count_vm
force_retire_all_vm
DELETE /clouds/CLOUD/vms/VMID
delete_vm_entry
PUT /clouds/CLOUD/vms/VMID?action=(reset_override_state|shutdown)
reset_override_state
shutdown_vm
POST /clouds/CLOUD/vms/VMID?action=force_retire
force_retire_vm
POST /users/USER?refresh=(job_proxy|vm_proxy)
refresh_job_proxy_user
refresh_vm_proxy_user
GET /cloud-aliases
list_cloud_alias
POST /cloud-aliases
cloud_alias_reload
GET /user-limits
list_user_limits
POST /user-limits
user_limit_reload
cloud_status
GET /
get_version
GET /cloud
get_cloud_resources
GET /cloud/config
get_cloud_config_values
GET /clusters
get_cluster_resources
GET /clusters.json
get_json_resource
GET /clusters/CLUSTER
get_cluster_info
GET /clusters/CLUSTER.json
get_json_cluster
GET /clusters/CLUSTER/vms
get_vm_stats
get_total_vms_cloud
GET /clusters/CLUSTER/vms/VMID
get_vm_info
GET /clusters/CLUSTER/vms/VMID.json
get_json_vm
GET /jobs?state=(new|sched|high|idle|running|complete|held)
get_newjobs
get_schedjobs
get_highjobs
get_idlejobs
get_runningjobs
get_completejobs
get_heldjobs
GET /jobs/JOBID
get_job
GET /jobs/JOBID.json
get_json_job
GET /vms[?metric=total|missing|startup_time|job_run_times]
get_vm_stats
get_total_vms
get_missing_vm_list
get_vm_startup_time
get_vm_job_run_times
GET /vms/resources
get_cluster_vm_resources
GET /job_pools.json
get_json_jobpool
GET /developer_info
get_developer_information
GET /ips
get_ips_munin
GET /diff-types
get_diff_types
GET /failures?type=(boot|image)
get_job_failure_reasons
get_image_failures
GET /shared-objs
check_shared_objs
GET /thread-heart-beats
get_thread_heart_beats
Server-side
On the server-side, web.py provides the HTTP server and request handling. web.py is a minimalist web framework that maps a set of URLs to view classes, and HTTP methods to view class instance methods. Replacing the XML-RPC server required modifying the imports, creating view classes and methods, moving the XML-RPC functions into the corresponding views, and replacing the code that started the
SimpleXMLRPCServer
with code that instantiates and starts a web.py server.
Typically a web.py application is served using
self.app = web.application(self.urls, globals())
self.app.run()
but this provides very little configurability. For instance, the port on which the server listens is taken from the first command line argument and cannot be specified otherwise. web.py embeds the WSGI server from
CherryPy
, and
app.run()
simply provides a convenient wrapper. Instead,
CherryPyWSGIServer
is used directly:
self.app = web.application(self.urls, globals())
self.server = web.wsgiserver.CherryPyWSGIServer(self.listen, self.app.wsgifunc(), server_name="localhost")
self.server.start()
Note that this does require using web.py's internal API and it is possible that this API could change. A possible alternative would be to embed
CherryPyWSGIServer
or add some other WSGI server as an additional dependency.
Client-side
On the client-side, requests provides the HTTP client and allows the simple replacement of XML-RPC calls with HTTP requests. The requests library is significantly simpler than Python's built-in HTTP library, urrlib. Making HTTP requests and handling the responses can be done in a single line of code. As an example, an XML-RPC request such as
print s.get_cluster_info(cli_options.cluster_name)
is replaced with
print requests.get(base_url + 'clusters/' + cli_options.cluster_name).text
where
base_url
gives the hostname and port (
http://localhost:8111
). The client code is otherwise unchanged, except that requests throws
requests.exceptions.ConnectionError
instead of
socket.error
.
Finally, requests automatically quotes/encodes URLs, parameters, and data, but since user input is being used directly in the URLs, additional quoting is performed. This allows any character to be used. For example, in the GET request above, if
cli_options.cluster_name
contains a
/
, as in
test/name
the following URL would be used:
Quoted:
http://localost:8111/clusters/test%2Fname
Unquoted:
http://localost:8111/clusters/test/name
The unquoted request would cause a 404 Not Found error, since no URL pattern matches this input. The quoted request is matched by
/clusters/([\w\%-]+)
Other problem characters include
?
and
#
, both of which are handled correctly.
Testing
Manual testing was performed locally and on a development server. In addition, a simple test script,
client_test
was written that runs each possible client command and logs the output.
--
ringda - 2016-01-08
Comments