Merge pull request #21 from deadc0de6/transformations

Transformations

README.md

@@ -59,18 +59,20 @@ why dotdrop rocks.
 
 * [Installation](#installation)
 * [Usage](#usage)
+* How to
 
-  * [Installing dotfiles](#installing-dotfiles)
+  * [Install dotfiles](#install-dotfiles)
-  * [Diffing your local dotfiles with dotdrop](#diffing-your-local-dotfiles-with-dotdrop)
+  * [Compare dotfiles](#compare-dotfiles)
-  * [Import new dotfiles](#import-new-dotfiles)
+  * [Import dotfiles](#import-dotfiles)
-  * [List the available profiles](#list-the-available-profiles)
+  * [List profiles](#list-profiles)
-  * [List configured dotfiles](#list-configured-dotfiles)
+  * [List dotfiles](#list-dotfiles)
-  * [Execute an action when deploying a dotfile](#execute-an-action-when-deploying-a-dotfile)
+  * [Use actions](#use-actions)
-  * [All dotfiles for a profile](#all-dotfiles-for-a-profile)
+  * [Use transformations](#use-transformations)
-  * [Include dotfiles from another profile](#include-dotfiles-from-another-profile)
   * [Update dotdrop](#update-dotdrop)
   * [Update dotfiles](#update-dotfiles)
+  * [Store sensitive dotfiles](#store-sensitive-dotfiles)
 
+* [Config](#config)
 * [Template](#template)
 * [Example](#example)
 * [People using dotdrop](#people-using-dotdrop)
@@ -78,11 +80,13 @@ why dotdrop rocks.
 # Installation
 
 There's two ways of installing and using dotdrop, either [as a submodule](#as-a-submodule)
-to your dotfiles git tree or system-wide [through pypi](#with-pypi).
+to your dotfiles git tree or system-wide [with pypi](#with-pypi).
 
 Having dotdrop as a submodule guarantees that anywhere your are cloning your dotfiles git tree
 from you'll have dotdrop shipped with it. It is the recommended way.
 
+Dotdrop is also available on aur: https://aur.archlinux.org/packages/dotdrop/
+
 ## As a submodule
 
 The following will create a repository for your dotfiles and
@@ -223,55 +227,7 @@ Options:
 For easy deployment the default profile used by dotdrop reflects the
 hostname of the host on which it runs.
 
-## Config file details
+## Install dotfiles
-
-The config file (defaults to *config.yaml*) is a yaml file containing
-the following entries:
-
-* **config** entry: contains settings for the deployment
-  * `backup`: create a backup of the dotfile in case it differs from the
-    one that will be installed by dotdrop
-  * `create`: create directory hierarchy when installing dotfiles if
-    it doesn't exist
-  * `dotpath`: path to the directory containing the dotfiles to be managed
-    by dotdrop (absolute path or relative to the config file location)
-
-* **dotfiles** entry: a list of dotfiles
-  * When `link` is true, dotdrop will create a symlink instead of copying. Template generation (as in [template](#template)) is not supported when `link` is true.
-  * `actions` contains a list of action keys that need to be defined in the **actions** entry below.
-```
-  <dotfile-key-name>:
-    dst: <where-this-file-is-deployed>
-    src: <filename-within-the-dotpath>
-    # Optional
-    link: <true|false>
-    actions:
-      - <action-key>
-```
-
-* **profiles** entry: a list of profiles with the different dotfiles that
-  need to be managed
-  * `dotfiles`: the dotfiles associated to this profile
-  * `include`: include all dotfiles from another profile (optional)
-
-```
-  <some-name-usually-the-hostname>:
-    dotfiles:
-    - <some-dotfile-key-name-defined-above>
-    - <some-other-dotfile-key-name>
-    - ...
-    # Optional
-    include:
-    - <some-other-profile>
-    - ...
-```
-
-* **actions** entry: a list of action
-```
-  <action-key>: <command-to-execute>
-```
-
-## Installing dotfiles
 
 Simply run
 ```bash
@@ -281,7 +237,7 @@ $ dotdrop.sh install
 Use the `--profile` switch to specify a profile if not using
 the host's hostname.
 
-## Diffing your local dotfiles with dotdrop
+## Compare dotfiles
 
 Compare local dotfiles with dotdrop's defined ones:
 ```bash
@@ -291,7 +247,7 @@ $ dotdrop.sh compare
 The diffing is done by diff in the backend, one can provide specific
 options to diff using the `-o` switch.
 
-## Import new dotfiles
+## Import dotfiles
 
 Dotdrop allows to import dotfiles directly from the
 filesystem. It will copy the dotfile and update the
@@ -300,10 +256,9 @@ config file automatically.
 For example to import `~/.xinitrc`
 ```bash
 $ dotdrop.sh import ~/.xinitrc
-
 ```
 
-## List the available profiles
+## List profiles
 
 ```bash
 $ dotdrop.sh list
@@ -313,7 +268,7 @@ Dotdrop allows to choose which profile to use
 with the *--profile* switch if you use something
 else than the default (the hostname).
 
-## List configured dotfiles
+## List dotfiles
 
 The following command lists the different dotfiles
 configured for a specific profile:
@@ -332,7 +287,7 @@ f_dunstrc (file: "config/dunst/dunstrc", link: False)
 	-> ~/.config/dunst/dunstrc
 ```
 
-## Execute an action when deploying a dotfile
+## Use actions
 
 It is sometimes useful to execute some kind of action
 when deploying a dotfile. For example let's consider
@@ -364,6 +319,143 @@ Thus when `f_vimrc` is installed, the command
 `vim +VundleClean! +VundleInstall +VundleInstall! +qall` will
 be executed.
 
+## Use transformations
+
+Transformation actions are used to transform a dotfile before it is
+installed. They work like [actions](#use-actions) but are executed before the
+dotfile is installed to transform the source.
+
+Transformation commands have two arguments:
+
+* **{0}** will be replaced with the dotfile to process
+* **{1}** will be replaced with a temporary file to store the result of the transformation
+
+A typical use-case for transformations is when the dotfile needs to be
+stored encrypted.
+
+Here's an example of part of a config file to use gpg encrypted dotfiles:
+```
+dotfiles:
+  f_secret:
+    dst: ~/.secret
+    src: secret
+    trans:
+      - gpg
+trans:
+  gpg: gpg2 -q --for-your-eyes-only --no-tty -d {0} > {1}
+```
+
+The above config allows to store the dotfile `~/.secret` encrypted in the *dotfiles*
+directory and uses gpg to decrypt it when install is run.
+
+Here's how to deploy the above solution:
+
+* import the clear dotfile (creates the correct entries in the config file)
+```
+./dotdrop.sh import ~/.secret
+```
+* encrypt the original dotfile 
+```
+<some-gpg-command> ~/.secret
+```
+* overwrite the dotfile with the encrypted version
+```
+cp <encrypted-version-of-secret> dotfiles/secret
+```
+* edit the config file and add the transformation to the dotfile
+* commit and push the changes
+
+## Update dotdrop
+
+If used as a submodule, update it with
+```bash
+$ git submodule foreach git pull origin master
+$ git add dotdrop
+$ git commit -m 'update dotdrop'
+$ git push
+```
+
+Through pypi:
+```bash
+$ sudo pip3 install dotdrop --upgrade
+```
+
+## Update dotfiles
+
+Dotfiles managed by dotdrop can be updated using the `update` command.
+There are two cases:
+
+  * the dotfile doesn't use [templating](#template): the new version of the dotfile is copied to the
+    *dotfiles* directory and overwrites the old version. If git is used to version the dotfiles stored
+    by dotdrop, the git command `diff` can be used to view the changes.
+  * the dotfile uses [templating](#template): the dotfile must be manually updated, the use of
+    the dotdrop command `compare` can be helpful to identify the changes to apply to the template.
+
+```
+$ dotdrop.sh update ~/.vimrc
+```
+
+## Store sensitive dotfiles
+
+Two solutions exist, the first one using an unversioned file (see [Environment variables](#environment-variables))
+and the second using transformations (see [Transformations](#use-transformations)).
+
+# Config
+
+The config file (defaults to *config.yaml*) is a yaml file containing
+the following entries:
+
+* **config** entry: contains settings for the deployment
+  * `backup`: create a backup of the dotfile in case it differs from the
+    one that will be installed by dotdrop
+  * `create`: create directory hierarchy when installing dotfiles if
+    it doesn't exist
+  * `dotpath`: path to the directory containing the dotfiles to be managed
+    by dotdrop (absolute path or relative to the config file location)
+
+* **dotfiles** entry: a list of dotfiles
+  * When `link` is true, dotdrop will create a symlink instead of copying. Template generation (as in [template](#template)) is not supported when `link` is true.
+  * `actions` contains a list of action keys that need to be defined in the **actions** entry below.
+  * `trans` contains a list of transformation keys that need to be defined in the **trans** entry below.
+```
+  <dotfile-key-name>:
+    dst: <where-this-file-is-deployed>
+    src: <filename-within-the-dotpath>
+    # Optional
+    link: <true|false>
+    actions:
+      - <action-key>
+    trans:
+      - <transformation-key>
+```
+
+* **profiles** entry: a list of profiles with the different dotfiles that
+  need to be managed
+  * `dotfiles`: the dotfiles associated to this profile
+  * `include`: include all dotfiles from another profile (optional)
+
+```
+  <some-name-usually-the-hostname>:
+    dotfiles:
+    - <some-dotfile-key-name-defined-above>
+    - <some-other-dotfile-key-name>
+    - ...
+    # Optional
+    include:
+    - <some-other-profile>
+    - ...
+```
+
+* **actions** entry: a list of action
+```
+  <action-key>: <command-to-execute>
+```
+
+* **trans** entry: a list of transformations
+```
+  <trans-key>: <command-to-execute>
+```
+
 ## All dotfiles for a profile
 
 To use all defined dotfiles for a profile, simply use
@@ -406,37 +498,6 @@ profiles:
 ```
 Here profile *host1* contains all the dotfiles defined for *host2* plus `f_xinitrc`.
 
-## Update dotdrop
-
-If used as a submodule, update it with
-```bash
-$ git submodule foreach git pull origin master
-$ git add dotdrop
-$ git commit -m 'update dotdrop'
-$ git push
-```
-
-Through pypi:
-```bash
-$ sudo pip3 install dotdrop --upgrade
-```
-
-
-## Update dotfiles
-
-Dotfiles managed by dotdrop can be updated using the `update` command.
-There are two cases:
-
-  * the dotfile doesn't use [templating](#template): the new version of the dotfile is copied to the
-    *dotfiles* directory and overwrites the old version. If git is used to version the dotfiles stored
-    by dotdrop, the git command `diff` can be used to view the changes.
-  * the dotfile uses [templating](#template): the dotfile must be manually updated, the use of
-    the dotdrop command `compare` can be helpful to identify the changes to apply to the template.
-
-```
-$ dotdrop.sh update ~/.vimrc
-```
-
 # Template
 
 Dotdrop leverage the power of [jinja2](http://jinja.pocoo.org/) to handle the
@@ -455,11 +516,10 @@ Note that dotdrop uses different delimiters than
 
 ## Available variables
 
-### Profile
+* `{{@@ profile @@}}` contains the profile provided to dotdrop.
+* `{{@@ env['MY_VAR'] @@}}` contains environment variables (see [Environment variables](#environment-variables))
 
-`{{@@ profile @@}}` contains the profile provided to dotdrop. Below example shows how it is used.
+## Environment variables
-
-### Environment variables
 
 It's possible to access environment variables inside the templates. This feature can be used like this:
 
@@ -481,7 +541,6 @@ pass="verysecurepassword"
 ```
 Of course, this file should not be tracked by git (put it in your `.gitignore`).
 
-
 Then you can invoke dotdrop with the help of an alias like that:
 ```
 ## when using dotdrop as a submodule
@@ -621,7 +680,7 @@ $ git push
 ```
 
 Otherwise, simply install it from pypi as explained [above](#with-pypi)
-and get rid of the submodule:
+and get rid of the submodule as shown below:
 
 * move to the dotfiles directory where dotdrop is used as a submodule
 ```bash

dotdrop/action.py

@@ -5,6 +5,7 @@ Represent an action in dotdrop
 """
 
 import subprocess
+import os
 
 # local imports
 from dotdrop.logger import Logger
@@ -18,11 +19,30 @@ class Action:
         self.log = Logger()
 
     def execute(self):
+        ret = 1
         self.log.sub('executing \"%s\"' % (self.action))
         try:
-            subprocess.call(self.action, shell=True)
+            ret = subprocess.call(self.action, shell=True)
         except KeyboardInterrupt:
             self.log.warn('action interrupted')
+        return ret == 0
+
+    def transform(self, arg0, arg1):
+        '''execute transformation with {0} and {1}
+        where {0} is the file to transform and
+        {1} is the result file'''
+        if os.path.exists(arg1):
+            msg = 'transformation destination exists: %s'
+            self.log.warn(msg % (arg1))
+            return False
+        ret = 1
+        cmd = self.action.format(arg0, arg1)
+        self.log.sub('transforming with \"%s\"' % (cmd))
+        try:
+            ret = subprocess.call(cmd, shell=True)
+        except KeyboardInterrupt:
+            self.log.warn('action interrupted')
+        return ret == 0
 
     def __str__(self):
         return 'key:%s -> \"%s\"' % (self.key, self.action)

dotdrop/config.py

@@ -18,6 +18,7 @@ class Cfg:
     key_config = 'config'
     key_dotfiles = 'dotfiles'
     key_actions = 'actions'
+    key_trans = 'trans'
     key_dotpath = 'dotpath'
     key_profiles = 'profiles'
     key_profiles_dots = 'dotfiles'
@@ -26,6 +27,7 @@ class Cfg:
     key_dotfiles_dst = 'dst'
     key_dotfiles_link = 'link'
     key_dotfiles_actions = 'actions'
+    key_dotfiles_trans = 'trans'
 
     def __init__(self, cfgpath):
         if not os.path.exists(cfgpath):
@@ -41,6 +43,8 @@ class Cfg:
         # not linked to content
         self.actions = {}
         # not linked to content
+        self.trans = {}
+        # not linked to content
         self.prodots = {}
         if not self._load_file():
             raise ValueError('config is not valid')
@@ -94,6 +98,12 @@ class Cfg:
                 for k, v in self.content[self.key_actions].items():
                     self.actions[k] = Action(k, v)
 
+        # parse all transformations
+        if self.key_trans in self.content:
+            if self.content[self.key_trans] is not None:
+                for k, v in self.content[self.key_trans].items():
+                    self.trans[k] = Action(k, v)
+
         # parse the profiles
         self.profiles = self.content[self.key_profiles]
         if self.profiles is None:
@@ -117,8 +127,17 @@ class Cfg:
             entries = v[self.key_dotfiles_actions] if \
                 self.key_dotfiles_actions in v else []
             actions = self._parse_actions(self.actions, entries)
-            self.dotfiles[k] = Dotfile(k, dst, src,
+            entries = v[self.key_dotfiles_trans] if \
-                                       link=link, actions=actions)
+                self.key_dotfiles_trans in v else []
+            trans = self._parse_actions(self.trans, entries)
+            if len(trans) > 0 and link:
+                msg = 'transformations disabled for \"%s\"' % (dst)
+                msg += ' as link is True' % (dst)
+                self.log.warn(msg)
+                trans = []
+            self.dotfiles[k] = Dotfile(k, dst, src, link=link,
+                                       actions=actions,
+                                       trans=trans)
 
         # assign dotfiles to each profile
         for k, v in self.profiles.items():

dotdrop/dotdrop.py

@@ -45,6 +45,7 @@ CUR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 LOG = Logger()
 HOSTNAME = os.uname()[1]
 TILD = '~'
+TRANS_SUFFIX = 'trans'
 
 BANNER = """     _       _      _
   __| | ___ | |_ __| |_ __ ___  _ __
@@ -101,7 +102,29 @@ def install(opts, conf):
         if hasattr(dotfile, 'link') and dotfile.link:
             r = inst.link(dotfile.src, dotfile.dst)
         else:
-            r = inst.install(t, opts['profile'], dotfile.src, dotfile.dst)
+            src = dotfile.src
+            tmp = None
+            if dotfile.trans:
+                tmp = '%s.%s' % (src, TRANS_SUFFIX)
+                err = False
+                for trans in dotfile.trans:
+                    s = os.path.join(opts['dotpath'], src)
+                    temp = os.path.join(opts['dotpath'], tmp)
+                    if not trans.transform(s, temp):
+                        msg = 'transformation \"%s\" failed for %s'
+                        LOG.err(msg % (trans.key, dotfile.key))
+                        err = True
+                        break
+                if err:
+                    if tmp and os.path.exists(tmp):
+                        remove(tmp)
+                    continue
+                src = tmp
+            r = inst.install(t, opts['profile'], src, dotfile.dst)
+            if tmp:
+                tmp = os.path.join(opts['dotpath'], tmp)
+                if os.path.exists(tmp):
+                    remove(tmp)
         if len(r) > 0 and len(dotfile.actions) > 0:
             # execute action
             for action in dotfile.actions:
@@ -140,6 +163,10 @@ def compare(opts, conf, tmp, focus=None):
         return ret
 
     for dotfile in selected:
+        if dotfile.trans:
+            msg = 'ignore %s as it uses transformation(s)'
+            LOG.log(msg % (dotfile.key))
+            continue
         same, diff = inst.compare(t, tmp, opts['profile'],
                                   dotfile.src, dotfile.dst,
                                   opts=opts['dopts'])

dotdrop/dotfile.py

@@ -7,7 +7,8 @@ represents a dotfile in dotdrop
 
 class Dotfile:
 
-    def __init__(self, key, dst, src, actions=[], link=False):
+    def __init__(self, key, dst, src,
+                 actions=[], trans=[], link=False):
         # key of dotfile in the config
         self.key = key
         # where to install this dotfile
@@ -18,6 +19,8 @@ class Dotfile:
         self.link = link
         # list of actions
         self.actions = actions
+        # list of transformations
+        self.trans = trans
 
     def __str__(self):
         return 'key:%s, src: %s, dst: %s, link: %s' % (self.key, self.src,

dotdrop/utils.py

@@ -38,6 +38,11 @@ def get_tmpdir():
     return tempfile.mkdtemp(prefix='dotdrop-')
 
 
+def get_tmpfile():
+    (fd, path) = tempfile.mkstemp(prefix='dotdrop-')
+    return path
+
+
 def remove(path):
     ''' Remove a file / directory / symlink '''
     if not os.path.exists(path):