usrlog

usrlog: Shell CLI REPL command logs in userspace (per virtualenv)

usrlog.sh

Source: https://github.com/westurner/dotfiles/blob/master/scripts/usrlog.sh

usrlog logs the most recent userspace interactive shell commandline CLI command to $_USRLOG after the shell command completes, with Bash PROMPT_COMMAND and ZSH precmd_functions (see: Caveats).

# setup bash
if [ -n "$BASH" ]; then
    PROMPT_COMMAND="_usrlog_writecmd; _usrlog_echo_title;"
fi

# setup zsh
if [ -n "$ZSH_VERSION" ]; then
    precmd_functions=(_usrlog_writecmd _usrlog_echo_title)
fi

Installation

usrlog.sh:

# source "${__DOTFILES}/etc/bash/30-bashrc.usrlog.sh  # {
source "${__DOTFILES}/scripts/usrlog.sh"              # }
echo "${_USRLOG}"$'\t'"${_TERM_ID}"         # dotfiles_status, ds, cls

Features

  • Initial Use Case: merge .bash_history and .zsh_history files

  • append each Bash or ZSH userspace commandstr to one file:

    _USRLOG="${HOME}/-usrlog.log"

  • append each Bash or ZSH userspace commandstr to one file per $VIRTUAL_ENV:

    _USRLOG="${VIRTUAL_ENV}/-usrlog.log"

  • add a session id to differentiate between multiple term instances:

    echo "${_TERM_ID}"
termid

  • generate relatively unique session ids by default when $_TERM_ID is unset:

    set_term_id [<name>]        # stid "#project/uri-fragment"
stid
st

  • include the $_TERM_ID in the shell prompt.

  • include VIRTUAL_ENV_NAME, TERM_ID, USER, HOSTNAME, and CWD in the window title (and PS1):

    ### usrlog window title
# (VIRTUAL_ENV_NAME) $_TERM_ID $USER@$HOSTNAME:/$CWD
(dotfiles) #2T515Vf25Ko user@host:~/-wrk/-ve27/dotfiles/src/dotfiles

  • include the previous command in the shell prompt PS1:

    ### PS1 shell prompt
# echo 'previous command'$'\t'"${PS1}"
(dotfiles) #2T515Vf25Ko user@host:~/-wrk/-ve27/dotfiles/src/dotfiles
$

  • [ ] BUG: escape tabs \t and newlines \r\n when setting _TERM_ID in order to support really any string (such as URI fragments):

    stid '#westurner/dotfiles/\tinstall-screencast'

usrlog venv integration

  • venv (virtualenvwrapper virtualenvs):

    echo "${_USRLOG}"        # _USRLOG=$HOME/-usrlog.log
echo "${HISTFILE}"       # HISTFILE=$HOME/.bash_history

workon_venv dotfiles     # _USRLOG=$VIRTUAL_ENV/-usrlog.log
                         # HISTFILE=$VIRTUAL_ENV/.bash_history
we dotfiles              # "
workon_conda dotfiles    # "
wec                      # "

# deactivating resets to __USRLOG
echo "${__USRLOG}  *"         # $HOME/-usrlog.log *
echo $_USRLOG                 # $VIRTUAL_ENV/-usrlog.log
deactivate; echo $_USRLOG     # $HOME/-usrlog.log

  • If defined, _usrlog_set_title calls _setup_venv_prompt (on every PROMPT_COMMAND invocation)

    ### scripts/usrlog.sh
function _usrlog_set_title {
    # _usrlog_set_title()  --  set xterm title
    export WINDOW_TITLE=${1:-"$_TERM_ID"}
    _usrlog_echo_title
    declare -f '_setup_venv_prompt' 2>&1 > /dev/null \
        && _setup_venv_prompt
}

    A _setup_venv_prompt is defined in https://github.com/westurner/dotfiles/blob/master/etc/bash/10-bashrc.venv.sh:

    ### etc/bash/10-bashrc.venv.sh
function _setup_venv_prompt { ... }
_setup_venv_prompt

  • _usrlog_grep_venv – find commands pertaining to venv

-usrlog.log format

  • Almost but not quite TSV (tab-separated values)
    • Maximum split (commands may contain additional unescaped tabs)
    • Initial “# ” in the date column
      • Because newlines
  • -usrlog.log format parsers:

Syntax

#\s\t                           # assert col[0] == '# '
<iso8601datewithtz_finish>\t
<_TERM_ID>\t                    # usrlog_set_id (stid)
<path>\t                        # path *after* command has run (see: prev)
<hist_n>\t                      # sometimes null (\s\s)
<iso8601datewithtz_start>\t
<HOSTNAME>\t                    # $HOSTNAME when _usrlog_set_HIST runs
<USER>\t                        # $USER when _usrlog_set_HIST runs
$$\t                            # cmd = split('\t$$\t', 1)[1]  # pyline
<cmd>                           # (history 1 || fc -l -1) | sedescape

-usrlog.log natural key

natural_key = (iso8601datewithtz_finish, $_TERM_ID, $HOSTNAME, $USER)

See: Caveats

usrlog.py

Source: https://github.com/westurner/dotfiles/blob/master/scripts/usrlog.py

usrlog.py is a commandline script for parsing -usrlog.log files.

Installation:

python "${__DOTFILES}/scripts/usrlog.py"
# echo "$PATH"              # dotfiles_status, ds, cls
cat $_USRLOG | usrlog.py --cmds -

Usage:

python usrlog.py --help

python usrlog.py --cmds $_USRLOG
cat $_USRLOG | tail -n 20 | usrlog.py --cmds -

Todo:

  • [ ] ENH: pandas.DataFrame.from_records (index=cols[1])
  • [ ] REF: update westurner/workhours integration

Caveats

  • A -usrlog.log is a text file log which commands are appended to, in userspace, with user permissions
    • Linux: see also: avc.log, audit.log
  • [ ] BUG: path is recorded after the command evaluates (see the previous command)
    • pass forward the ending path from the previous command (PATH_PREV) in PROMPT_COMMAND
    • initial PATH_PREV (at shell startup):
      • see the previous command (in the -usrlog.log file)
  • [ ] BUG: usrlog.sh only records the last line of commands containing newlines
    • history 1 | parse | template | tee $_USRLOG
  • Sometimes, hist_n is not set (e.g. when n > HISTSIZE), but the tabs should still be there.
  • Sometimes, iso8601datewithtz_start is not set (e.g. when HISTTIMEFORMAT is not set), but the tabs should still be there.
  • I still have logs with the old .usrlog formats to parse (~^2015), which is why usrlog.py is far more complex than usrlog.sh _usrlog_parse_cmds (cat $_USRLOG | ugp):
    • [ ] REF: write migration scripts for oldstyle logs
      • workaround: grep -C $_USRLOG
      • workaround: grep -C $(lsusrlogs) | ugp
      • workaround: grep -C $(lsusrlogs) | usrlog.py --cmds

Ideas

  • ENH: prompt square:

    # prev_cmd
# date_start
# date_finish [timedelta]
# (VIRTUAL_ENV_NAME) $_TERM_ID $USER@$HOSTNAME:/$CWD
$
    • PERF: time delta overhead
    • UBY: minimize n rows