Merge pull request #319 from deadc0de6/userinput

Userinput

docs/config-details.md

@@ -287,6 +287,37 @@ variables:
 
 They have the same properties as [Variables](config.md#variables).
 
+## Entry uservariables
+
+If you want to manually enter variables values, you can use the
+`uservariables` entry. Each variable will be prompted to the user.
+
+For example
+```yaml
+uservariables:
+  emailvar: "email"
+```
+
+will prompt the user to enter a value for the variable `emailvar`:
+```
+Please provide the value for "email":
+```
+
+And store the entered text as the value for the variable `email`.
+The variable can then be used as any other [variables](config.md#variables).
+
+`uservariables` are eventually saved to `uservariables.yaml` (relatively to the
+config file).
+This allow to use the following construct to prompt once for some specific variables and
+then store them in a file. You might also want to add `uservariables.yaml` to your `.gitignore`.
+```yaml
+uservariables:
+  emailvar: "email"
+config:
+  import_variables:
+    - uservariables.yaml:optional
+```
+
 ## Entry profile variables
 
 Profile variables will take precedence over globally defined variables.

docs/config-format.md

@@ -45,8 +45,8 @@ The **dotfiles** entry (mandatory) contains a list of dotfiles managed by dotdro
 
 Entry    | Description
 -------- | -------------
-`dst` | where this dotfile needs to be deployed (dotfile with empty `dst` are ignored and considered installed, can use `variables` and `dynvariables`, make sure to quote)
+`dst` | where this dotfile needs to be deployed (dotfile with empty `dst` are ignored and considered installed, can use `variables`, make sure to quote)
-`src` | dotfile path within the `dotpath` (dotfile with empty `src` are ignored and considered installed, can use `variables` and `dynvariables`, make sure to quote)
+`src` | dotfile path within the `dotpath` (dotfile with empty `src` are ignored and considered installed, can use `variables`, make sure to quote)
 `link` | define how this dotfile is installed. Possible values: *nolink*, *link*, *link_children* (see [Symlinking dotfiles](config.md#symlink-dotfiles)) (defaults to value of `link_dotfile_default`)
 `actions` | list of action keys that need to be defined in the **actions** entry below (see [actions](config-details.md#entry-actions))
 `chmod` | defines the file permissions in octal notation to apply during installation (see [permissions](config.md#permissions))
@@ -177,3 +177,14 @@ The **dynvariables** entry (optional) contains a list of interpreted variables
 dynvariables:
   <variable-name>: <shell-oneliner>
 ```
+
+## uservariables entry
+
+The **uservariables** entry (optional) contains a list of variables
+to be queried to the user for their values.
+(see [User variables](config-details.md#entry-uservariables))
+
+```yaml
+uservariables:
+  <variable-name>: <prompt>
+```

docs/config.md

@@ -39,6 +39,7 @@ Following variables are available in the config files:
 
 * [variables defined in the config](config-details.md#entry-variables)
 * [interpreted variables defined in the config](config-details.md#entry-dynvariables)
+* [user variables defined in the config](config-details.md#entry-uservariables)
 * [profile variables defined in the config](config-details.md#entry-profile-variables)
 * environment variables: `{{@@ env['MY_VAR'] @@}}`
 * dotdrop header: `{{@@ header() @@}}` (see [Dotdrop header](templating.md#dotdrop-header))
@@ -60,6 +61,8 @@ Here are some rules on the use of variables in configs:
 * profile `(dyn)variables` take precedence over profile's included `(dyn)variables`
 * external/imported `(dyn)variables` take precedence over
   `(dyn)variables` defined inside the main config file
+* [user variables](config-details.md#entry-uservariables) are ignored if
+  any other variable with the same key is defined
 
 ## Permissions
 

docs/howto/howto.md

@@ -43,3 +43,7 @@
 ## Symlink dotfiles
 
 [Symlink dotfiles](symlink-dotfiles.md)
+
+## Prompt user for variables
+
+[Prompt user for variables](prompt-user-for-variables.md)

docs/howto/prompt-user-for-variables.md

@@ -0,0 +1,26 @@
+# Prompt user for variables
+
+With the use of [uservariables](../config-details.md#entry-uservariables),
+one can define specific variables that need to be initially filled in manually
+by the user on first run.
+
+The provided values are then automatically saved by dotdrop to `uservariables.yaml`,
+which can be included in the main config as a file from which variables are imported
+using [import_variables](../config-details.md#entry-import_variables).
+
+Let's say for example that you want to provide manually the email value
+on new hosts you deploy your dotfiles to.
+
+You'd add the following elements to your config:
+```yaml
+uservariables:
+  emailvar: "email"
+config:
+  import_variables:
+    - uservariables.yaml:optional
+```
+
+On first run, the `emailvar` is prompted to the user and then saved
+to `uservariables.yaml`. Since this file is imported, the value for
+`emailvar` will automatically be filled in without prompting the
+user on subsequent calls.

dotdrop/cfg_aggregator.py

@@ -206,10 +206,11 @@ class CfgAggregator:
     # parsing
     ########################################################
 
-    def _load(self):
+    def _load(self, reloading=False):
         """load lower level config"""
         self.cfgyaml = CfgYaml(self.path,
                                self.profile_key,
+                               reloading=reloading,
                                debug=self.debug)
 
         # settings
@@ -361,7 +362,7 @@ class CfgAggregator:
         self.log.dbg('reloading config')
         olddebug = self.debug
         self.debug = False
-        self._load()
+        self._load(reloading=True)
         self.debug = olddebug
 
     @classmethod

dotdrop/cfg_yaml.py

@@ -33,7 +33,7 @@ from dotdrop.settings import Settings
 from dotdrop.logger import Logger
 from dotdrop.templategen import Templategen
 from dotdrop.linktypes import LinkTypes
-from dotdrop.utils import shellrun, uniq_list
+from dotdrop.utils import shellrun, uniq_list, userinput
 from dotdrop.exceptions import YamlException, UndefinedException
 
 
@@ -50,10 +50,13 @@ class CfgYaml:
     key_trans_w = 'trans_write'
     key_variables = 'variables'
     key_dvariables = 'dynvariables'
+    key_uvariables = 'uservariables'
 
     action_pre = 'pre'
     action_post = 'post'
 
+    save_uservariables_name = 'uservariables{}.yaml'
+
     # profiles/dotfiles entries
     key_dotfile_src = 'src'
     key_dotfile_dst = 'dst'
@@ -98,16 +101,19 @@ class CfgYaml:
     allowed_link_val = [lnk_nolink, lnk_link, lnk_children]
     top_entries = [key_dotfiles, key_settings, key_profiles]
 
-    def __init__(self, path, profile=None, addprofiles=None, debug=False):
+    def __init__(self, path, profile=None, addprofiles=None,
+                 reloading=False, debug=False):
         """
         config parser
         @path: config file path
         @profile: the selected profile
         @addprofiles: included profiles
+        @reloading: true when reloading
         @debug: debug flag
         """
         self._path = os.path.abspath(path)
         self._profile = profile
+        self._reloading = reloading
         self._debug = debug
         self._log = Logger(debug=self._debug)
         # config needs to be written
@@ -249,6 +255,12 @@ class CfgYaml:
         # template dotfiles entries
         self._template_dotfiles_entries()
 
+        # parse the "uservariables" block
+        uvariables = self._parse_blk_uservariables(self._yaml_dict,
+                                                   self.variables)
+        self._add_variables(uvariables, template=False, prio=False)
+
+        # end of parsing
         if self._debug:
             self._dbg('########### {} ###########'.format('final config'))
             self._debug_entries()
@@ -428,7 +440,7 @@ class CfgYaml:
         if self._debug:
             self._dbg('saving to {}'.format(self._path))
         try:
-            with open(self._path, 'w') as file:
+            with open(self._path, 'w', encoding='utf8') as file:
                 self._yaml_dump(content, file)
         except Exception as exc:
             self._log.err(exc)
@@ -565,6 +577,41 @@ class CfgYaml:
             self._debug_dict('dynvariables block', dvariables)
         return dvariables
 
+    def _parse_blk_uservariables(self, dic, current):
+        """parse the "uservariables" block"""
+        uvariables = self._get_entry(dic,
+                                     self.key_uvariables,
+                                     mandatory=False)
+
+        uvars = {}
+        if not self._reloading and uvariables:
+            try:
+                for name, prompt in uvariables.items():
+                    if name in current:
+                        # ignore if already defined
+                        if self._debug:
+                            self._dbg('ignore uservariables {}'.format(name))
+                        continue
+                    content = userinput(prompt, debug=self._debug)
+                    uvars[name] = content
+            except KeyboardInterrupt as exc:
+                raise YamlException('interrupted') from exc
+
+            if uvars:
+                uvars = uvars.copy()
+        if self._debug:
+            self._debug_dict('uservariables block', uvars)
+
+        # save uservariables
+        if uvars:
+            try:
+                self._save_uservariables(uvars)
+            except YamlException:
+                # ignore
+                pass
+
+        return uvars
+
     ########################################################
     # parsing helpers
     ########################################################
@@ -1079,7 +1126,7 @@ class CfgYaml:
         if self._debug:
             self._dbg('----------start:{}----------'.format(path))
             cfg = '\n'
-            with open(path, 'r') as file:
+            with open(path, 'r', encoding='utf8') as file:
                 for line in file:
                     cfg += line
             self._dbg(cfg.rstrip())
@@ -1124,7 +1171,7 @@ class CfgYaml:
     @classmethod
     def _yaml_load(cls, path):
         """load from yaml"""
-        with open(path, 'r') as file:
+        with open(path, 'r', encoding='utf8') as file:
             data = yaml()
             data.typ = 'rt'
             content = data.load(file)
@@ -1282,7 +1329,7 @@ class CfgYaml:
 
         # the included profiles
         inc_profiles = []
-        if self._profile and self._profile in self.profiles.keys():
+        if self._profile and self._profile in self.profiles:
             pentry = self.profiles.get(self._profile)
             inc_profiles = pentry.get(self.key_profile_include, [])
 
@@ -1355,7 +1402,7 @@ class CfgYaml:
         - Checks for path existence, taking in account fatal_not_found.
         This method always returns a list containing only absolute paths
         existing on the filesystem. If the input is not a glob, the list
-        contains at most one element, otheriwse it could hold more.
+        contains at most one element, otherwise it could hold more.
         """
         path, fatal_not_found = self._parse_extended_import_path(path_entry)
         path = self._norm_path(path)
@@ -1515,3 +1562,34 @@ class CfgYaml:
     def _dbg(self, content):
         pre = os.path.basename(self._path)
         self._log.dbg('[{}] {}'.format(pre, content))
+
+    def _save_uservariables(self, uvars):
+        """save uservariables to file"""
+        parent = os.path.dirname(self._path)
+        # find a unique path
+        path = None
+        cnt = 0
+        while True:
+            if cnt == 0:
+                name = self.save_uservariables_name.format('')
+            else:
+                name = self.save_uservariables_name.format('-{}'.format(cnt))
+            cnt += 1
+
+            path = os.path.join(parent, name)
+            if not os.path.exists(path):
+                break
+
+        # save the config
+        content = {'variables': uvars}
+        try:
+            if self._debug:
+                self._dbg('saving uservariables values to {}'.format(path))
+            with open(path, 'w', encoding='utf8') as file:
+                self._yaml_dump(content, file)
+        except Exception as exc:
+            # self._log.err(exc)
+            err = 'error saving uservariables to {}'.format(path)
+            self._log.err(err)
+            raise YamlException(err) from exc
+        self._log.log('uservariables values saved to {}'.format(path))

dotdrop/templategen.py

@@ -183,7 +183,7 @@ class Templategen:
         path = os.path.normpath(path)
         if not os.path.exists(path):
             raise TemplateNotFound(path)
-        with open(path, 'r') as file:
+        with open(path, 'r', encoding='utf8') as file:
             content = file.read()
         return content
 

dotdrop/utils.py

@@ -69,6 +69,20 @@ def shellrun(cmd, debug=False):
     return ret == 0, out
 
 
+def userinput(prompt, debug=False):
+    """
+    get user input
+    return user input
+    """
+    if debug:
+        LOG.dbg('get user input for \"{}\"'.format(prompt), force=True)
+    pre = 'Please provide the value for \"{}\": '.format(prompt)
+    res = input(pre)
+    if debug:
+        LOG.dbg('user input result: {}'.format(res), force=True)
+    return res
+
+
 def fastdiff(left, right):
     """fast compare files and returns True if different"""
     return not filecmp.cmp(left, right, shallow=False)

tests-ng/uservariables.sh

@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+# author: deadc0de6 (https://github.com/deadc0de6)
+# Copyright (c) 2021, deadc0de6
+#
+# test user variables from yaml file
+# returns 1 in case of error
+#
+
+# exit on first error
+set -e
+
+# all this crap to get current path
+rl="readlink -f"
+if ! ${rl} "${0}" >/dev/null 2>&1; then
+  rl="realpath"
+
+  if ! hash ${rl}; then
+    echo "\"${rl}\" not found !" && exit 1
+  fi
+fi
+cur=$(dirname "$(${rl} "${0}")")
+
+#hash dotdrop >/dev/null 2>&1
+#[ "$?" != "0" ] && echo "install dotdrop to run tests" && exit 1
+
+#echo "called with ${1}"
+
+# dotdrop path can be pass as argument
+ddpath="${cur}/../"
+[ "${1}" != "" ] && ddpath="${1}"
+[ ! -d ${ddpath} ] && echo "ddpath \"${ddpath}\" is not a directory" && exit 1
+
+export PYTHONPATH="${ddpath}:${PYTHONPATH}"
+bin="python3 -m dotdrop.dotdrop"
+hash coverage 2>/dev/null && bin="coverage run -a --source=dotdrop -m dotdrop.dotdrop" || true
+
+echo "dotdrop path: ${ddpath}"
+echo "pythonpath: ${PYTHONPATH}"
+
+# get the helpers
+source ${cur}/helpers
+
+echo -e "$(tput setaf 6)==> RUNNING $(basename $BASH_SOURCE) <==$(tput sgr0)"
+
+################################################################
+# this is the test
+################################################################
+
+# the dotfile source
+tmps=`mktemp -d --suffix='-dotdrop-tests' || mktemp -d`
+mkdir -p ${tmps}/dotfiles
+# the dotfile destination
+tmpd=`mktemp -d --suffix='-dotdrop-tests' || mktemp -d`
+#echo "dotfile destination: ${tmpd}"
+
+# create the config file
+cfg="${tmps}/config.yaml"
+
+cat > ${cfg} << _EOF
+config:
+  backup: true
+  create: true
+  dotpath: dotfiles
+variables:
+  var4: "variables_var4"
+dynvariables:
+  var3: "echo dynvariables_var3"
+uservariables:
+  var1: "var1"
+  var2: "var2"
+  var3: "var3"
+  var4: "var4"
+dotfiles:
+  f_abc:
+    dst: ${tmpd}/abc
+    src: abc
+profiles:
+  p1:
+    dotfiles:
+    - f_abc
+_EOF
+#cat ${cfg}
+
+# create the dotfile
+echo "var1: {{@@ var1 @@}}" > ${tmps}/dotfiles/abc
+echo "var2: {{@@ var2 @@}}" >> ${tmps}/dotfiles/abc
+echo "var3: {{@@ var3 @@}}" >> ${tmps}/dotfiles/abc
+echo "var4: {{@@ var4 @@}}" >> ${tmps}/dotfiles/abc
+
+# install
+cd ${ddpath} | echo -e 'var1contentxxx\nvar2contentyyy\nvar3\nvar4\n' | ${bin} install -f -c ${cfg} -p p1 -V
+
+cat ${tmpd}/abc
+
+grep '^var1: var1contentxxx$' ${tmpd}/abc >/dev/null
+grep '^var2: var2contentyyy$' ${tmpd}/abc >/dev/null
+grep '^var3: dynvariables_var3$' ${tmpd}/abc >/dev/null
+grep '^var4: variables_var4$' ${tmpd}/abc >/dev/null
+
+## CLEANING
+rm -rf ${tmps} ${tmpd} ${scr}
+
+echo "OK"
+exit 0

tests.sh

@@ -6,6 +6,14 @@
 #set -ev
 set -e
 
+# versions
+echo "pylint version:"
+pylint --version
+echo "pycodestyle version:"
+pycodestyle --version
+echo "pyflakes version:"
+pyflakes --version
+
 # PEP8 tests
 which pycodestyle >/dev/null 2>&1
 [ "$?" != "0" ] && echo "Install pycodestyle" && exit 1
@@ -30,6 +38,7 @@ pylint \
   --disable=R0912 \
   --disable=R0911 \
   --disable=R1732 \
+  --disable=C0209 \
   dotdrop/
 
 # retrieve the nosetests binary
@@ -38,6 +47,8 @@ which ${nosebin} >/dev/null 2>&1
 [ "$?" != "0" ] && nosebin="nosetests3"
 which ${nosebin} >/dev/null 2>&1
 [ "$?" != "0" ] && echo "Install nosetests" && exit 1
+echo "nose version:"
+${nosebin} --version
 
 # do not print debugs when running tests (faster)
 export DOTDROP_FORCE_NODEBUG=yes