1
0
mirror of https://github.com/deadc0de6/dotdrop.git synced 2026-02-04 13:56:44 +00:00

add import and trans{_r,_w}

This commit is contained in:
deadc0de6
2022-06-05 08:47:01 +02:00
committed by deadc0de
parent 97917c2f70
commit dc68277ab8
11 changed files with 222 additions and 114 deletions

View File

@@ -16,14 +16,14 @@ There are two types of transformations available:
* **Read transformations**: used to transform dotfiles before they are installed ([config](config-config.md) key `trans_read`)
* Used for commands `install` and `compare`
* They have two arguments:
* They have two mandatory arguments:
* **{0}** will be replaced with the dotfile to process
* **{1}** will be replaced with a temporary file to store the result of the transformation
* This Happens **before** the dotfile is templated (see [templating](templating.md))
* **Write transformations**: used to transform files before updating a dotfile ([config](config-config.md) key `trans_write`)
* Used for command `update`
* They have two arguments:
* Used for command `update` and `import`
* They have two mandatory arguments:
* **{0}** will be replaced with the file path to update the dotfile with
* **{1}** will be replaced with a temporary file to store the result of the transformation

View File

@@ -1,70 +1,109 @@
# Handle secrets
Two solutions exist, the first one using an unversioned file (see [Environment variables](../templating.md#environment-variables))
and the second using transformations (see [Store encrypted dotfiles](#store-encrypted-dotfiles)).
* [Using environment variables](#using-environment-variables)
* [Store encrypted dotfiles using GPG](#store-encrypted-dotfiles-using-gpg)
* [GPG examples](#gpg-examples)
* [Store encrypted dotfiles](#store-encrypted-dotfiles)
* [Load passphrase from file](#load-passphrase-from-file)
## Using environment variables
## Store encrypted dotfiles
For example, you can have an `.env` file in the directory where your `config.yaml` lies:
```bash
## Some secrets
pass="verysecurepassword"
```
Here's an example of part of a config file to use gpg encrypted dotfiles:
If this file contains secrets that should not be tracked by Git,
put it in your `.gitignore`.
You can then invoke dotdrop with the help of an alias
```bash
# when dotdrop is installed as a submodule
alias dotdrop='eval $(grep -v "^#" ~/dotfiles/.env) ~/dotfiles/dotdrop.sh'
# when dotdrop is installed from package
alias dotdrop='eval $(grep -v "^#" ~/dotfiles/.env) /usr/bin/dotdrop --cfg=~/dotfiles/config.yaml'
```
The above aliases load all the variables from `~/dotfiles/.env`
(while omitting lines starting with `#`) before calling dotdrop.
Defined variables can then be used [in the config](../config-file.md#template-config-entries)
or [for templating dotfiles](../templating.md)
For more see [the doc on environment variables](../templating.md#environment-variables).
## Store encrypted dotfiles using GPG
First you need to define the encryption/decryption methods, for example
```yaml
dotfiles:
f_secret:
dst: ~/.secret
src: secret
trans_read: _gpg
variables:
keyid: "11223344"
trans_read:
_gpg: gpg2 -q --for-your-eyes-only --no-tty -d {0} > {1}
_decrypt: "gpg -q --for-your-eyes-only--no-tty -d {0} > {1}"
trans_write:
_encrypt: "gpg -q -r {{@@ keyid @@}} --armor --no-tty -o {1} -e {0}"
```
The above config allows to store the dotfile `~/.secret` encrypted in the *dotpath*
directory and uses gpg to decrypt it when `install` is run.
Here's how to deploy the above solution:
* Import the clear dotfile (what creates the correct entries in the config file):
You can then import your dotfile and specify the transformations to apply/associate.
```bash
$ dotdrop import ~/.secret
dotdrop import --transw=_encrypt --transr=_decrypt ~/.secret
```
* Encrypt the original dotfile:
```bash
$ <some-gpg-command> ~/.secret
```
* Overwrite the dotfile with the encrypted version:
```bash
$ cp <encrypted-version-of-secret> dotfiles/secret
```
* Edit the config file and add the transformation to the dotfile
(as shown in the example above)
* Commit and push the changes
Now whenever you install/compare your dotfile, the `_decrypt` transformation will be executed
to get the clear version of the file.
When updating the `_encrypt` transformation will transform the file to store it encrypted.
See [transformations](../config-transformations.md).
## Load passphrase from file
## gpg examples
Using GPG keys:
```yaml
variables:
keyid: "11223344"
trans_read:
_decrypt: "gpg -q --for-your-eyes-only--no-tty -d {0} > {1}"
trans_write:
_encrypt: "gpg -q -r {{@@ keyid @@}} --armor --no-tty -o {1} -e {0}"
```
Passphrase is stored in a environement variable:
```yaml
trans_read:
_decrypt: "echo {{@@ env['THE_KEY'] @@}} | gpg -q --batch --yes --for-your-eyes-only --passphrase-fd 0 --no-tty -d {0} > {1}"
trans_write:
_encrypt: "echo {{@@ env['THE_KEY'] @@}} | gpg -q --batch --yes --passphrase-fd 0 --no-tty -o {1} -c {0}"
```
Passphrase is stored as a variable:
```yaml
variables:
gpg_password: "some password"
trans_read:
_decrypt: "echo {{@@ gpg_password @@}} | gpg -q --batch --yes --for-your-eyes-only --passphrase-fd 0 --no-tty -d {0} > {1}"
trans_write:
_encrypt: "echo {{@@ gpg_password @@}} | gpg -q --batch --yes --passphrase-fd 0 --no-tty -o {1} -c {0}"
```
Passphrase is retrieved using a script:
```yaml
variables:
dynvariables:
gpg_password: "./get-password.sh"
trans_read:
_gpg: "gpg2 --batch --yes --passphrase-file <({{@@ gpg_password @@}}) -q --for-your-eyes-only --no-tty -d {0} > {1}"
_decrypt: "echo {{@@ gpg_password @@}} | gpg -q --batch --yes --for-your-eyes-only --passphrase-fd 0 --no-tty -d {0} > {1}"
trans_write:
_encrypt: "echo {{@@ gpg_password @@}} | gpg -q --batch --yes --passphrase-fd 0 --no-tty -o {1} -c {0}"
```
Passphrase is stored in a file directly:
Passphrase is stored in a file:
```yaml
variables:
gpg_password_file: "/tmp/the-password"
dynvariables:
gpg_password: "cat {{@@ gpg_password_file @@}}"
trans_read:
_gpg: "gpg2 --batch --yes --passphrase-file <(cat {{@@ gpg_password_file @@}}) -q --for-your-eyes-only --no-tty -d {0} > {1}"
_decrypt: "echo {{@@ gpg_password @@}} | gpg -q --batch --yes --for-your-eyes-only --passphrase-fd 0 --no-tty -d {0} > {1}"
trans_write:
_encrypt: "echo {{@@ gpg_password @@}} | gpg -q --batch --yes --passphrase-fd 0 --no-tty -o {1} -c {0}"
```
See [transformations](../config-transformations.md).
See also [transformations](../config-transformations.md).

View File

@@ -3,34 +3,27 @@
This is an example of how to use transformations (`trans_read` and `trans_write`) to store
compressed directories and deploy them with dotdrop.
Config file:
Start by defining the transformations:
```yaml
trans_read:
uncompress: "mkdir -p {1} && tar -xf {0} -C {1}"
trans_write:
compress: "tar -cf {1} -C {0} ."
config:
backup: true
create: true
dotpath: dotfiles
dotfiles:
d_somedir:
dst: ~/.somedir
src: somedir
trans_read: uncompress
trans_write: compress
profiles:
p1:
dotfiles:
- d_somedir
```
The *read* transformation `uncompress` is used to execute the below command before deploying the dotfile (where `{0}` is the source and `{1}` the destination):
Then import the directory by specifying which transformations to apply/associate:
```bash
dotdrop import --transw=compress --transr=uncompress ~/.somedir
```
The *read* transformation `uncompress` is used to execute the below command before installing/comparing the dotfile (where `{0}` is the source and `{1}` the destination):
```bash
mkdir -p {1} && tar -xf {0} -C {1}
```
And the *write* transformation `compress` is run when updating the dotfile directory by compressing it (where `{0}` is the source and `{1}` the destination):
```
```bash
tar -cf {1} -C {0} .
```
See [transformations](../config-transformations.md).

View File

@@ -75,27 +75,7 @@ It's possible to access environment variables inside the templates:
This allows for storing host-specific properties and/or secrets in environment variables.
It is recommended to use `variables` (see [config variables](config-file.md#variables))
instead of environment variables unless these contain sensitive information that
shouldn't be versioned in Git.
For example, you can have an `.env` file in the directory where your `config.yaml` lies:
```
## Some secrets
pass="verysecurepassword"
```
If this file contains secrets that should not be tracked by Git,
put it in your `.gitignore`.
You can then invoke dotdrop with the help of an alias
```bash
# when dotdrop is installed as a submodule
alias dotdrop='eval $(grep -v "^#" ~/dotfiles/.env) ~/dotfiles/dotdrop.sh'
# when dotdrop is installed from pypi or aur
alias dotdrop='eval $(grep -v "^#" ~/dotfiles/.env) /usr/bin/dotdrop --cfg=~/dotfiles/config.yaml'
```
The above aliases load all the variables from `~/dotfiles/.env`
(while omitting lines starting with `#`) before calling dotdrop.
shouldn't be versioned in Git (see [handle secrets doc](howto/sensitive-dotfiles.md)).
## Template methods

View File

@@ -56,28 +56,36 @@ class CfgAggregator:
"""remove this dotfile from this profile"""
return self.cfgyaml.del_dotfile_from_profile(dotfile.key, profile.key)
def new_dotfile(self, src, dst, link, chmod=None):
def new_dotfile(self, src, dst, link, chmod=None,
trans_read=None, trans_write=None):
"""
import a new dotfile
@src: path in dotpath
@dst: path in FS
@link: LinkType
@chmod: file permission
@trans_read: read transformation
@trans_write: write transformation
"""
dst = self.path_to_dotfile_dst(dst)
dotfile = self.get_dotfile_by_src_dst(src, dst)
if not dotfile:
dotfile = self._create_new_dotfile(src, dst, link, chmod=chmod)
# add the dotfile
dotfile = self._create_new_dotfile(src, dst, link, chmod=chmod,
trans_read=trans_read,
trans_write=trans_write)
if not dotfile:
return False
# add to profile
key = dotfile.key
ret = self.cfgyaml.add_dotfile_to_profile(key, self.profile_key)
if ret:
msg = 'new dotfile {} to profile {}'
self.log.dbg(msg.format(key, self.profile_key))
# save the config and reload it
if ret:
self._save_and_reload()
return ret
@@ -205,15 +213,26 @@ class CfgAggregator:
# accessors for public methods
########################################################
def _create_new_dotfile(self, src, dst, link, chmod=None):
def _create_new_dotfile(self, src, dst, link, chmod=None,
trans_read=None, trans_write=None):
"""create a new dotfile"""
# get a new dotfile with a unique key
key = self._get_new_dotfile_key(dst)
self.log.dbg('new dotfile key: {}'.format(key))
# add the dotfile
if not self.cfgyaml.add_dotfile(key, src, dst, link, chmod=chmod):
trans_r_key = trans_w_key = None
if trans_read:
trans_r_key = trans_read.key
if trans_write:
trans_w_key = trans_write.key
if not self.cfgyaml.add_dotfile(key, src, dst, link,
chmod=chmod,
trans_r_key=trans_r_key,
trans_w_key=trans_w_key):
return None
return Dotfile(key, dst, src)
return Dotfile(key, dst, src,
trans_r=trans_read,
trans_w=trans_write)
########################################################
# parsing
@@ -282,7 +301,7 @@ class CfgAggregator:
# patch trans_w/trans_r in dotfiles
self._patch_keys_to_objs(self.dotfiles,
"trans_r",
self._get_trans_w_args(self._get_trans_r),
self._get_trans_w_args(self.get_trans_r),
islist=False)
self._patch_keys_to_objs(self.dotfiles,
"trans_w",
@@ -453,7 +472,7 @@ class CfgAggregator:
return trans
return getit
def _get_trans_r(self, key):
def get_trans_r(self, key):
"""return the trans_r with this key"""
try:
return next(x for x in self.trans_r if x.key == key)

View File

@@ -394,7 +394,8 @@ class CfgYaml:
self._dirty = True
return True
def add_dotfile(self, key, src, dst, link, chmod=None):
def add_dotfile(self, key, src, dst, link, chmod=None,
trans_r_key=None, trans_w_key=None):
"""add a new dotfile"""
if key in self.dotfiles.keys():
return False
@@ -404,10 +405,15 @@ class CfgYaml:
self._dbg('new dotfile dst: {}'.format(dst))
self._dbg('new dotfile link: {}'.format(link))
self._dbg('new dotfile chmod: {}'.format(chmod))
self._dbg('new dotfile trans_r: {}'.format(trans_r_key))
self._dbg('new dotfile trans_w: {}'.format(trans_w_key))
# create the dotfile dict
df_dict = {
self.key_dotfile_src: src,
self.key_dotfile_dst: dst,
}
# link
dfl = self.settings[self.key_settings_link_dotfile_default]
if str(link) != dfl:
@@ -417,6 +423,15 @@ class CfgYaml:
if chmod:
df_dict[self.key_dotfile_chmod] = str(format(chmod, 'o'))
# trans_r/trans_w
if trans_r_key:
df_dict[self.key_trans_r] = str(trans_r_key)
if trans_w_key:
df_dict[self.key_trans_w] = str(trans_w_key)
if self._debug:
self._dbg('dotfile dict: {}'.format(df_dict))
# add to global dict
self._yaml_dict[self.key_dotfiles][key] = df_dict
self._dirty = True

View File

@@ -529,6 +529,7 @@ def cmd_importer(opts):
paths = opts.import_path
importer = Importer(opts.profile, opts.conf,
opts.dotpath, opts.diff_command,
opts.variables,
dry=opts.dry, safe=opts.safe,
debug=opts.debug,
keepdot=opts.keepdot,
@@ -538,7 +539,8 @@ def cmd_importer(opts):
tmpret = importer.import_path(path, import_as=opts.import_as,
import_link=opts.import_link,
import_mode=opts.import_mode,
import_transw=opts.import_transw)
import_transw=opts.import_transw,
import_transr=opts.import_transr)
if tmpret < 0:
ret = False
elif tmpret > 0:

View File

@@ -15,19 +15,21 @@ from dotdrop.utils import strip_home, get_default_file_perms, \
get_unique_tmp_name, removepath
from dotdrop.linktypes import LinkTypes
from dotdrop.comparator import Comparator
from dotdrop.templategen import Templategen
class Importer:
"""dotfile importer"""
def __init__(self, profile, conf, dotpath, diff_cmd,
dry=False, safe=True, debug=False,
variables, dry=False, safe=True, debug=False,
keepdot=True, ignore=None):
"""constructor
@profile: the selected profile
@conf: configuration manager
@dotpath: dotfiles dotpath
@diff_cmd: diff command to use
@variables: dictionary of variables for the templates
@dry: simulate
@safe: ask for overwrite if True
@debug: enable debug
@@ -38,19 +40,25 @@ class Importer:
self.conf = conf
self.dotpath = dotpath
self.diff_cmd = diff_cmd
self.variables = variables
self.dry = dry
self.safe = safe
self.debug = debug
self.keepdot = keepdot
self.ignore = ignore or []
self.templater = Templategen(variables=self.variables,
base=self.dotpath,
debug=self.debug)
self.umask = get_umask()
self.log = Logger(debug=self.debug)
def import_path(self, path, import_as=None,
import_link=LinkTypes.NOLINK,
import_mode=False,
import_transw=""):
import_transw="",
import_transr=""):
"""
import a dotfile pointed by path
returns:
@@ -66,18 +74,22 @@ class Importer:
# check transw if any
trans_write = None
trans_read = None
if import_transw:
trans_write = self.conf.get_trans_w(import_transw)
if import_transr:
trans_read = self.conf.get_trans_r(import_transr)
return self._import(path, import_as=import_as,
import_link=import_link,
import_mode=import_mode,
trans_write=trans_write)
trans_write=trans_write,
trans_read=trans_read)
def _import(self, path, import_as=None,
import_link=LinkTypes.NOLINK,
import_mode=False,
trans_write=None):
trans_write=None, trans_read=None):
"""
import path
returns:
@@ -135,13 +147,14 @@ class Importer:
if not self._import_file(src, dst, trans_write=trans_write):
return -1
# TODO add trans_write
# TODO add trans_read too
return self._import_in_config(path, src, dst, perm, linktype,
import_mode)
import_mode,
trans_w=trans_write,
trans_r=trans_read)
def _import_in_config(self, path, src, dst, perm,
linktype, import_mode):
linktype, import_mode,
trans_r=None, trans_w=None):
"""
import path
returns:
@@ -159,7 +172,9 @@ class Importer:
chmod = perm
# add file to config file
retconf = self.conf.new_dotfile(src, dst, linktype, chmod=chmod)
retconf = self.conf.new_dotfile(src, dst, linktype, chmod=chmod,
trans_read=trans_r,
trans_write=trans_w)
if not retconf:
self.log.warn('\"{}\" ignored during import'.format(path))
return 0
@@ -292,7 +307,8 @@ class Importer:
return path
self.log.dbg('executing write transformation {}'.format(trans))
tmp = get_unique_tmp_name()
if not trans.transform(path, tmp, debug=self.debug):
if not trans.transform(path, tmp, debug=self.debug,
templater=self.templater):
msg = 'transformation \"{}\" failed for {}'
self.log.err(msg.format(trans.key, path))
if os.path.exists(tmp):

View File

@@ -60,7 +60,8 @@ Usage:
dotdrop install [-VbtfndDaW] [-c <path>] [-p <profile>]
[-w <nb>] [<key>...]
dotdrop import [-Vbdfm] [-c <path>] [-p <profile>] [-i <pattern>...]
[-l <link>] [-S <key>] [-s <path>] <path>...
[--transr=<key>] [--transw=<key>]
[-l <link>] [-s <path>] <path>...
dotdrop compare [-LVbz] [-c <path>] [-p <profile>]
[-w <nb>] [-C <file>...] [-i <pattern>...]
dotdrop update [-VbfdkPz] [-c <path>] [-p <profile>]
@@ -90,7 +91,8 @@ Options:
-p --profile=<profile> Specify the profile to use [default: {}].
-P --show-patch Provide a one-liner to manually patch template.
-s --as=<path> Import as a different path from actual path.
-S --transw=<key> Apply trans_write key on import.
--transr=<key> Associate trans_read key on import.
--transw=<key> Apply trans_write key on import.
-t --temp Install to a temporary directory for review.
-T --template Only template dotfiles.
-V --verbose Be verbose.
@@ -275,6 +277,7 @@ class Options(AttrMonitor):
self.import_ignore.append('*{}'.format(self.install_backup_suffix))
self.import_ignore = uniq_list(self.import_ignore)
self.import_transw = self.args['--transw']
self.import_transr = self.args['--transr']
def _apply_args_update(self):
"""update specifics"""

View File

@@ -63,11 +63,13 @@ cfg="${tmps}/config.yaml"
cat > ${cfg} << _EOF
trans_read:
base64: cat {0} | base64 -d > {1}
uncompress: mkdir -p {1} && tar -xf {0} -C {1}
base64: "cat {0} | base64 -d > {1}"
decompress: "mkdir -p {1} && tar -xf {0} -C {1}"
decrypt: "echo {{@@ profile @@}} | gpg -q --batch --yes --passphrase-fd 0 --no-tty -d {0} > {1}"
trans_write:
base64: cat {0} | base64 > {1}
compress: tar -cf {1} -C {0} .
base64: "cat {0} | base64 > {1}"
compress: "tar -cf {1} -C {0} ."
encrypt: "echo {{@@ profile @@}} | gpg -q --batch --yes --passphrase-fd 0 --no-tty -o {1} -c {0}"
config:
backup: true
create: true
@@ -80,25 +82,30 @@ _EOF
# tokens
token="test-base64"
tokend="compressed archive"
tokenenc="encrypted"
# create the dotfiles
echo ${token} > ${tmpd}/abc
mkdir -p ${tmpd}/def/a
echo ${tokend} > ${tmpd}/def/a/file
echo ${tokenenc} > ${tmpd}/ghi
###########################
# test import
###########################
echo "[+] run import"
# import file
cd ${ddpath} | ${bin} import -f -c ${cfg} -p p1 -b -V -S base64 ${tmpd}/abc
# import directory
cd ${ddpath} | ${bin} import -f -c ${cfg} -p p1 -b -V -S compress ${tmpd}/def
# import file (to base64)
cd ${ddpath} | ${bin} import -f -c ${cfg} -p p1 -b -V --transw=base64 --transr=base64 ${tmpd}/abc
# import directory (to compress)
cd ${ddpath} | ${bin} import -f -c ${cfg} -p p1 -b -V --transw=compress --transr=decompress ${tmpd}/def
# import file (to encrypt)
cd ${ddpath} | ${bin} import -f -c ${cfg} -p p1 -b -V --transw=encrypt --transr=decrypt ${tmpd}/ghi
# check file imported in dotpath
[ ! -e ${tmps}/dotfiles/${tmpd}/abc ] && echo "abc does not exist" && exit 1
[ ! -e ${tmps}/dotfiles/${tmpd}/def ] && echo "def does not exist" && exit 1
[ ! -e ${tmps}/dotfiles/${tmpd}/ghi ] && echo "ghi does not exist" && exit 1
# check content in dotpath
echo "checking content"
@@ -110,17 +117,49 @@ file ${tmps}/dotfiles/${tmpd}/def | grep -i 'tar'
tar -cf ${tmps}/test-def -C ${tmpd}/def .
diff ${tmps}/dotfiles/${tmpd}/def ${tmps}/test-def
file ${tmps}/dotfiles/${tmpd}/ghi | grep -i 'gpg symmetrically encrypted data'
echo p1 | gpg -q --batch --yes --passphrase-fd 0 --no-tty -d ${tmps}/dotfiles/${tmpd}/ghi > ${tmps}/test-ghi
diff ${tmps}/test-ghi ${tmpd}/ghi
# check is imported in config
echo "checking imported in config"
cd ${ddpath} | ${bin} -p p1 -c ${cfg} files
cd ${ddpath} | ${bin} -p p1 -c ${cfg} files | grep '^f_abc'
cd ${ddpath} | ${bin} -p p1 -c ${cfg} files | grep '^d_def'
cd ${ddpath} | ${bin} -p p1 -c ${cfg} files | grep '^f_ghi'
# check has trans_write in config
echo "checking trans_write is set in config"
echo "--------------"
cat ${cfg}
cat ${cfg} | grep -m 1 -A 3 'f_abc' | grep 'trans_write: base64'
cat ${cfg} | grep -m 1 -A 3 'd_def' | grep 'trans_write: compress'
echo "--------------"
cat ${cfg} | grep -m 1 -A 4 'f_abc' | grep 'trans_write: base64'
cat ${cfg} | grep -m 1 -A 4 'd_def' | grep 'trans_write: compress'
cat ${cfg} | grep -m 1 -A 4 'f_ghi' | grep 'trans_write: encrypt'
cat ${cfg} | grep -m 1 -A 4 'f_abc' | grep 'trans_read: base64'
cat ${cfg} | grep -m 1 -A 4 'd_def' | grep 'trans_read: decompress'
cat ${cfg} | grep -m 1 -A 4 'f_ghi' | grep 'trans_read: decrypt'
# install these
rm ${tmpd}/abc
rm -r ${tmpd}/def
rm ${tmpd}/ghi
cd ${ddpath} | ${bin} install -f -c ${cfg} -p p1 -b -V
# test exist
[ ! -e ${tmpd}/abc ] && exit 1
[ ! -d ${tmpd}/def/a ] && exit 1
[ ! -e ${tmpd}/def/a/file ] && exit 1
[ ! -e ${tmpd}/ghi ] && exit 1
# test content
cat ${tmpd}/abc
cat ${tmpd}/abc | grep "${token}"
cat ${tmpd}/def/a/file
cat ${tmpd}/def/a/file | grep "${tokend}"
cat ${tmpd}/ghi | grep "${tokenenc}"
echo "OK"
exit 0

View File

@@ -137,6 +137,8 @@ def _fake_args():
args['--preserve-mode'] = False
args['--ignore-missing'] = False
args['--workdir-clear'] = False
args['--transw'] = ''
args['--transr'] = ''
# cmds
args['profiles'] = False
args['files'] = False