Tags:
create new tag
view all tags
-- KevinCasteels - 2015-06-02

CloudInit yaml merging

The following describes how to merge multiple .yaml configuration files together when submitting as user data to CloudInit

Prerequisites

Mime Multi Part Archives In User Data

Sometimes it is required to provide multiple .yaml files as user data to cloudinit during a VM install. If the different .yaml files include the same keywords, for example 'write_files', then by default previous declarations of the keyword will be overwritten by the final occurrence. To illustrate, say we have two .yaml files which both use the keyword 'run_cmd':

#cloud-config (1)
run_cmd:
  - bash1
  - bash2

#cloud-config (2)
run_cmd:
  - bash3
  - bash4

This is the default way cloudinit merges these keywords and lists:

#cloud-config (merged)
run_cmd:
  - bash3
  - bash4 

By default, the second incidence of the keyword 'run_cmd' has over written the first. In this situation we would prefer the lists be appended, as follows:

#cloud-config (merged)
run_cmd:
  - bash1
  - bash2
  - bash3
  - bash4

To specify the way merging keyword lists is done, cloudinit allows a new keyword to be included in the .yaml files. These keywords are 'merge_how' and 'merge_type', and are searched for by cloudinit in that order. If these keywords are used multiple times in different .yaml files, the first occurrence is used and the following ones ignored.

The format for using the merging keywords is:

classname1(option1,option2)+classname2(option3,option4)....

There appear to be three 'classnames' available, and these are

list()
dict()
str()  

These classnames can accept different options. I have scoured the cloudinit source code and have found the following options:

append
prepend
recurse_array
recurse_list
recurse_str
allow_delete
replace
no_replace

The function of some of these seem obvious, others do not. Unfortunately I have been unable to find documentation describing these anywhere. This page will be updated as we use these classes and options and better understand how they work.

The following are some examples of how 'merge_how' and 'merge_type' can be used:

merge_how: 'dict(recurse_array,no_replace)+list(append)'
merge_how: 'dict(recurse_list)+list(append)'
merge_how: 'dict(recurse_array,no_replace)+list(prepend)'
merge_how: 'dict(allow_delete,no_replace)+list()'
merge_how: 'dict(replace)+list(append)'
merge_how: 'dict(recurse_str)+str(append)'
merge_how: 'dict(recurse_array)+list(recurse_list,recurse_str)+str()'
merge_how: "dict(recurse_array)+list(append)"
merge_how: "dict(allow_delete,recurse_array)+list(recurse_array,append)"
merge_type: 'list(append)+dict(recurse_array)+str()'

The final example,

merge_type: 'list(append)+dict(recurse_array)+str()'
has been found to correctly append keyword lists and is a good choice for merging .yaml files in CloudInit.

IMPORTANT: The merge_type directive must be included in each .yaml file.

For more information see:

http://cloudinit.readthedocs.org/en/latest/topics/merging.html#overview

https://github.com/number5/cloud-init/search?utf8=%E2%9C%93&q=merge

Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | More topic actions
Topic revision: r4 - 2015-10-08 - kcasteels
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback