Tools¶
See Also: https://westurner.org/wiki/projects#tools
Packages¶
A software package is an archive of files with a manifest that lists the files included. Often, the manifest contains file checksums and a signature.
Many packaging tools make a distinction between source and/or binary packages.
Some packaging tools provide configuration options for:
- Scripts to run when packaging
- Scripts to run at install time
- Scripts to run at uninstal time
- Patches to apply to the “vanilla” source tree, as might be obtained from a version control repository
There is a package maintainer whose responsibilities include:
- Testing new upstream releases
- Vetting changes from release to release
- Repackaging upstream releases
- Signing new package releases
Packaging lag refers to how long it takes a package maintainer to repackage upstream releases for the target platform(s).
Apt¶
APT (“Advanced Packaging Tool”) is the core of Debian package management.
An APT package repository serves Dpkg packages.
An APT package repository can be accessed from a local filesystem or over a network protocol (“apt transports”) like HTTP, HTTPS, RSYNC, FTP, and BitTorrent.
An example of APT usage (e.g. to maintain an updated Ubuntu Linux system):
apt-get update
apt-get upgrade
apt-get dist-upgrade
apt-cache show bash
apt-get install bash
apt-get --help
man apt-get
man sources.list
Bower¶
Bower is “a package manager for the web” (JavaScript packages) built on NPM.
DEB¶
DEB is the Debian software package format.
DEB packages are built with Dpkg and often hosted in an Apt package repository.
Dpkg¶
Dpkg is a collection of tools for creating and working with DEB packages.
NPM¶
NPM is a JavaScript package manager created for Node.js.
Bower builds on NPM.
NuGet¶
- Package Repositories (chocolatey):
- Linux/Mac/Windows: No / No / Yes
Ports¶
Sources and Makefiles designed to compile software packages for particular distributions’ kernel and standard libraries on a particular platform.
RPM¶
Install with
rpm
,yum
Build with tools like
rpmbuild
andfpm
Python: build with
bdist_rpm
,fpm
List contents:
less ~/path/to/local.rpm # requires lesspipe to be configured
Package Repositories (yum):
- Local: directories of packages and metadata
- Network: HTTP, HTTPS, RSYNC, FTP
Python Packages¶
A Python Package is a collection of source code and package data files.
- Python packages have dependencies: they depend on other packages
- Python packages can be served from a package index
- PyPI is the community Python Package Index
- A Python package is an archive of files
(
.zip
(.egg
,.whl
),.tar
,.tar.gz
,) containing asetup.py
file containing a version string and metadata that is meant for distribution. - An source dist (
sdist
) package contains source code (every file listed in or matching a pattern in aMANIFEST.in
text file). - A binary dist (
bdist
,bdist_egg
,bdist_wheel
) is derived from an sdist and may be compiled and named for a specific platform. - sdists and bdists are defined by a
setup.py
file which contains a call to adistutils.setup()
orsetuptools.setup()
function. - The arguments to the
setup.py
function are things likeversion
,author
,author_email
, andhomepage
; in addition to package dependency strings required for the package to work (install_requires
), for tests to run (tests_require
), and for optional things to work (extras_require
). - A package dependency string can specify an exact version (
==
) or a greater-than (>=
) or less-than (<=
) requirement for each package. - Package names are looked up from an index server (
--index
), such as PyPI, and or an HTML page (--find-links
) containing URLs containing package names, version strings, and platform strings. easy_install
(Setuptools) and Pip can install packages from: the local filesystem, a remote index server, or a local index server.easy_install
andpip
read theinstall_requires
(andextras_require
) attributes ofsetup.py
files contained in packages in order to resolve a dependency graph (which can contain cycles) and install necessary packages.
Distuils¶
Distutils is a collection of tools for common packaging needs.
Distutils is included in the Python standard library.
Setuptools¶
Setuptools is a Python package for working with other Python Packages.
Setuptools builds upon Distuils
Setuptools is widely implemented
Most Python packages are installed by setuptools (by Pip)
Setuptools can be installed by downloading
ez_setup.py
and then runningpython ez_setup.py
; or, setuptools can be installed with a system package manager (apt, yum)Setuptools installs a script called
easy_install
which can be used to install packages from the local filesystem, a remote index server, a local index server, or an HTML pageeasy_install pip
installs Pip from PyPILike
easy_install
, Pip installs python packages, with a number of additional configuration optionsSetuptools can build RPM and DEB packages from python packages, with some extra configuration:
python setup.py bdist_rpm --help python setup.py --command-packages=stdeb.command bdist_deb --help
Pip¶
Pip is a tool for installing, upgrading, and uninstalling Python packages.
pip help
pip help install
pip --version
sudo apt-get install python-pip
pip install --upgrade pip
pip install libcloud
pip install -r requirements.txt
pip uninstall libcloud
- Pip stands upon Distuils and Setuptools.
- Pip retrieves, installs, upgrades, and uninstalls packages.
- Pip can list installed packages with
pip freeze
(andpip list
). - Pip can install packages as ‘editable’ packages (
pip install -e
) from version control repository URLs which must begin withvcs+
, end with#egg=<usuallythepackagename>
, and may contain an@vcstag
tag (such as a branch name or a version tag). - Pip installs packages as editable by first
cloning (or checking out) the code to
./src
(or${VIRTUAL_ENV}/src
if working in a Virtualenv) and then runningsetup.py develop
. - Pip configuration is in
${HOME}/.pip/pip.conf
. - Pip can maintain a local cache of downloaded packages, which can lessen the load on package servers during testing.
- Pip skips reinstallation if a package requirement is already satisfied.
- Pip requires the
--upgrade
and/or--force-reinstall
options to be added to thepip install
command in order to upgrade or reinstall. - At the time of this writing, the latest stable pip version is
1.5.6
.
Warning
With Python 2, pip is preferable to
Setuptools‘s easy_install
because pip installs backports.ssl_match_hostname
in order to validate HTTPS
certificates
(by making sure that the certificate hostname matches the hostname
from which the DNS resolved to).
Cloning packages from source repositories over ssh://
or https://
,
either manually or with pip install -e
avoids this concern.
There is also a tool called Peep which
requires considered-good SHA256 checksums to be specified
for every dependency listed in a requirements.txt
file.
For more information, see: http://legacy.python.org/dev/peps/pep-0476/#python-versions
- Pip Requirements File
Plaintext list of packages and package URIs to install.
Requirements files may contain version specifiers (
pip >= 1.5
)Pip installs Pip Requirement Files:
pip install -r requirements.txt pip install --upgrade -r requirements.txt pip install --upgrade --user --force-reinstall -r requirements.txt
An example
requirements.txt
file:# install pip from the default index (PyPI) pip --index=https://pypi.python.org/simple --upgrade pip # Install pip 1.5 or greater from PyPI pip >= 1.5 # Git clone and install pip as an editable develop egg -e git+https://github.com/pypa/pip@1.5.X#egg=pip # Install a source distribution release from PyPI # and check the MD5 checksum in the URL https://pypi.python.org/packages/source/p/pip/pip-1.5.5.tar.gz#md5=7520581ba0687dec1ce85bd15496537b # Install a source distribution release from Warehouse https://warehouse.python.org/packages/source/p/pip/pip-1.5.5.tar.gz # Install an additional requirements.txt file -r requirements/more-requirements.txt
Peep¶
Peep works just like Pip, but requires SHA256
checksum hashes
to be specified for each package in requirements.txt
file.
PyPI¶
PyPI is the Python Package Index.
Warehouse¶
Warehouse is the “Next Generation Python Package Repository”.
All packages uploaded to PyPI are also available from Warehouse.
Wheel¶
- Wheel is a newer, PEP-based standard (
.whl
) with a different metadata format, the ability to specify (JSON) digital signatures for a package within the package, and a number of additional speed and platform-consistency advantages. - Wheels can be uploaded to PyPI.
- Wheels are generally faster than traditional Python packages.
Packages available as wheels are listed at http://pythonwheels.com/.
Conda¶
- Conda installs packages written in any language; especially Python
conda skeleton
can automatically create conda packages both fromPyPI
and fromCPAN
(Perl)- Conda was originally created for the Anaconda Python Distribution, which installs packages written in Python, R, JavaScript, Ruby, C, Fortran
- Conda (and Anaconda) packages are hosted by
https://binstar.org,
which hosts free public and paid private Conda packages.
- Anaconda Server is an internal “Private, Secure Package Repository” that “supports over 100 different repositories, including PyPI, CRAN, conda, and the Anaconda repository.”
To create a fresh conda env:
# Python 2.7
conda env create -n exmpl2 --yes python readline pip
# conda install ipython-notebook
# Python 3.X
conda env create -n exmpl3 --yes python3 readline pip
Work on a conda env:
source activate exmpl2
conda list
source deactivate
RubyGems¶
- RubyGems installs Ruby Gems
Yum¶
Yum is a tool for installing, upgrading, and uninstalling RPM packages.
Version Control Systems¶
Version Control Systems (VCS) — or Revision Control Systems (RCS) — are designed to solve various problems in change management.
- VCS store code in a repository.
- Changes to one or more files are called changesets, commits, or revisions
- Changesets are comitted or checked into to a repository.
- Changesets are checked out from a repository
- Many/most VCS differentiate between the repository and a working directory, which is currently checked out to a specific changeset identified by a revision identifier; possibly with uncommitted local changes.
- A branch is forked from a line of development and then merged back in.
- Most projects designate a main line of development referred to as a trunk, master, or default branch.
- Many projects work with feature and release branches, which, ideally, eventually converge by being merged back into trunk. (see: HubFlow for an excellent example of branching)
- Traditional VCS are centralized on a single point-of-failure.
- Some VCS have a concept of locking to prevent multiple peoples’ changes from colliding
- Distributed Version Control Systems (DVCS) (can) clone all revisions of every branch of a repository every time. *
- DVCS changesets are pushed to a different repository
- DVCS changesets are pulled from another repository into a local clone or copy of a repository
- Teams working with DVCS often designate a central repository hosted by a project forge service like SourceForge, GNU Savannah, GitHub, or BitBucket.
- Contributors send patches which build upon a specific revision, which can be applied by a maintainer with commit access permissions.
- Contributors fork a new branch from a specific revision, commit changes, and then send a pull request, which can be applied by a maintainer with commit access permissions.
CVS¶
CVS (cvs
) is a centralized version control system (VCS) written in C.
CVS predates most/many other VCS.
Subversion¶
Apache Subversion (svn
) is a centralized revision control system (VCS)
written in C.
To checkout a revision of a repository with svn
:
svn co http://svn.apache.org/repos/asf/subversion/trunk subversion
Bazaar¶
GNU Bazaar (bzr
) is a distributed revision control system (DVCS, RCS, VCS)
written in Python and C.
http://launchpad.net hosts Bazaar repositories;
with special support from the bzr
tool in the form of lp:
urls
like lp:bzr
.
To clone a repository with bzr
:
bzr branch lp:bzr
Git¶
Git (git
) is a distributed version control system for tracking a branching
and merging repository of file revisions written in C (DVCS, VCS,
RCS).
To clone a repository with git
:
git clone https://github.com/git/git
HubFlow¶
HubFlow is a fork of GitFlow that adds extremely useful commands for working with Git and GitHub.
HubFlow is a named branch workflow with mostly-automated merges between branches.
Branch names are configurable; the defaults are as follows:
Branch Name | Description (and Code Labels) |
master |
Stable trunk (latest release) |
develop |
Development main line |
feature/<name> |
New features for the next release (e.g. ENH , PRF ) |
hotfix/<name> |
Fixes to merge to both master and develop
(e.g. BUG , TST , DOC ) |
release/<name> |
In-progress release branches (e.g. RLS ) |
Creating a new release with Git and HubFlow:
git clone ssh://git@github.com/westurner/dotfiles
# git checkout master
git hf init
## Update versiontag in .git/config to prefix release tags with 'v'
# [hubflow "prefix"]
# feature = feature/
# release = release/
# hotfix = hotfix/
# support = support/
# versiontag = v
#
git hf feature start print_hello_world
## commit, commit, commit
git hf feature finish print_hello_world
git hf release start 0.1.0
## commit (e.g. update version in setup.py, release notes)
git hf release finish 0.1.0
The GitFlow HubFlow illustrations are very helpful for visualizing and understanding any DVCS workflow: https://datasift.github.io/gitflow/IntroducingGitFlow.html.
Mercurial¶
Mercurial (hg
) is a distributed revision control system
written in Python and C (DVCS, VCS, RCS).
To clone a repository with hg
:
hg clone http://selenic.com/hg
Anaconda¶
Anaconda is a maintained distribution of many popular Python Packages.
Anaconda works with Conda packages.
Note
https://en.wikipedia.org/wiki/Anaconda_(installer) (1999) is the installer for RPM-based Linux distributions; which is also written in Python (and C).
Awk¶
AWK is a pattern programming language for matching and transforming text.
Bash¶
Bash, the Bourne-again shell.
type bash
bash --help
help help
help type
apropos bash
info bash
man bash
Designed to work with unix command outputs and return codes
Functions
Portability: sh (sh, bash, dash, zsh) shell scripts are mostly compatible
Logging:
set -x # print commands and arguments set -v # print source
Bash Configuration:
/etc/profile
/etc/bash.bashrc
/etc/profile.d/*.sh
${HOME}/.profile /etc/skel/.profile # PATH=+$HOME/bin # umask
${HOME}/.bash_profile # empty. preempts .profile
Linux/Mac/Windows: Almost Always / Bash 3.2 / Cygwin/Mingwin
Binutils¶
GNU Binutils are a set of utilities for working with assembly and binary.
GCC utilizes GNU Binutils to compile the GNU/Linux kernel and userspace.
GAS, the GNU Assembler (as
) assembles ASM code for linking by
the GNU linker (ld
).
C¶
C is a third-generation programming language which affords relatively low-level machine access while providing helpful abstractions.
The GNU/Linux kernel is written in C
and often compiled by GCC or Clang
for a particular architecture (see: man uname
)
Libc libraries are written in C.
Almost all of the projects linked here, at some point, utilize code written in C.
C++¶
C++ is a third-generation programming language which adds object orientation and a standard library to C.
- C++ is an ISO specification: C++98, C++03, C++11 (C++0x), C++14, [ C++17 ]
Clang¶
Clang is a compiler front end for C, C++, and Objective C/++.
Compiz¶
Compiz is a window compositing layer for X11 which adds lots of cool and productivity-enhancing visual capabilities.
CoreOS¶
CoreOS is a Linux distribution for highly available distributed computing.
CoreOS schedules redundant Docker images with fleet and systemd according to configuration stored in etcd, a key-value store with a D-Bus interface.
Docker¶
Docker is an OS virtualization project which utilizes Linux LXC Containers to partition process workloads all running under one kernel.
Limitations
- Writing to /etc/hosts: https://github.com/docker/docker/issues/2267
- Apt-get upgrade: https://github.com/docker/docker/issues/3934
Docutils¶
Docutils is a text processing system which ‘parses” ReStructuredText lightweight markup language into a doctree which it serializes into HTML, LaTeX, man-pages, Open Document files, XML, and a number of other formats.
Fortran¶
Fortran (or FORTRAN) is a third-generation programming language frequently used for mathematical and scientific computing.
Filesystem Hierarchy Standard¶
The Filesystem Hierarchy Standard is a well-worn industry-supported system file naming structure.
Ubuntu and Virtualenv implement a Filesystem Hierarchy.
Docker layers filesystem hierarchies with aufs and now also btrfs subvolumes.
GCC¶
The GNU Compiler Collection started as a Free and Open Source compiler for C.
There are now GCC frontends for many languages, including C++, Fortran, Java, and Go.
Gnome¶
Go¶
Go is a statically-typed C-based third generation language.
Grep¶
Grep is a UXIX CLI utility for pattern-based text matching.
Htop¶
I3wm¶
I3wm is a tiling window manager for X11 (Linux) with extremely-configurable Vim-like keyboard shortcuts.
IPython¶
IPython is an interactive REPL and distributed computation framework written in Python.
An IPython notebook file (.ipynb
) is a
JSON document containing input and output
for a linear sequence of cells;
which can be exported to many output formats (e.g. HTML, RST, LaTeX, PDF);
and edited through the web with
IPython Notebook.
1 + 1
x = 1+1
print("1 + 1 = %d" (x))
# IPython
? # help
%lsmagic
%<tab> # list magic commands and aliases
%logstart? # help for the %logstart magic command
%logstart -o logoutput.log.py # log input and output to a file
import json
json? # print(json.__doc__)
json?? # print(inspect.getsource(json))
# IPython shell
!cat ./README.rst; echo $PWD # run shell commands
lines = !ls -al # capture shell command output
print(lines[0:])
%run -i -t example.py # run a script with timing info,
# in the local namespace
%run -d example.py # run a script with pdb
%pdb on # automatically run pdb on Exception
Note
IPython notebook runs code and shell commands as the user the process is running as, on a remote or local machine.
IPython notebook supports more than 20 different languages.
Reproducible SciPy Stack IPython / Jupyter Notebook servers implement best practices like process isolation and privilege separation:
- https://registry.hub.docker.com/repos/ipython/
- https://registry.hub.docker.com/repos/jupyter/
- https://registry.hub.docker.com/u/jupyter/tmpnb/
IPython / Jupyter Notebook Viewer (nbviewer
)
is an application for serving read-only
versions of notebooks which have HTTP URLs.
Java¶
Java is a third-generation programming language which is
compiled into code that runs in a virtual machine
(JVM
) written in C for many different operating systems.
JavaScript¶
JavaScript is a third-generation programming language designed to run in an interpreter; now specified as ECMAScript.
All major web browsers support Javascript.
Client-side (web) applications can be written in Javascript.
Server-side (web) applications can be written in Javascript, often with Node.js and NPM packages.
Note
Java and JavaScript are two distinctly different languages and developer ecosystems.
JSON¶
JSON is an object representation in JavaScript syntax which is now supported by libraries for many language.
A list of objects with key
and value
attributes in JSON syntax:
[
{ "key": "language", "value": "Javascript" },
{ "key": "version", "value": 1 },
{ "key": "example", "value": true },
]
Machine-generated JSON is often not very readable, because it doesn’t contain extra spaces or newlines. The Python JSON library contains a utility for parsing and indenting (“prettifying”) JSON from the commandline
cat example.json | python -m json.tool
JSON-LD¶
JSON-LD is a web standard for Linked Data in JSON.
An example from the JSON-LD Playground (http://goo.gl/xxZ410):
{
"@context": {
"gr": "http://purl.org/goodrelations/v1#",
"pto": "http://www.productontology.org/id/",
"foaf": "http://xmlns.com/foaf/0.1/",
"xsd": "http://www.w3.org/2001/XMLSchema#",
"foaf:page": {
"@type": "@id"
},
"gr:acceptedPaymentMethods": {
"@type": "@id"
},
"gr:hasBusinessFunction": {
"@type": "@id"
},
"gr:hasCurrencyValue": {
"@type": "xsd:float"
}
},
"@id": "http://example.org/cars/for-sale#tesla",
"@type": "gr:Offering",
"gr:name": "Used Tesla Roadster",
"gr:description": "Need to sell fast and furiously",
"gr:hasBusinessFunction": "gr:Sell",
"gr:acceptedPaymentMethods": "gr:Cash",
"gr:hasPriceSpecification": {
"gr:hasCurrencyValue": "85000",
"gr:hasCurrency": "USD"
},
"gr:includes": {
"@type": [
"gr:Individual",
"pto:Vehicle"
],
"gr:name": "Tesla Roadster",
"foaf:page": "http://www.teslamotors.com/roadster"
}
}
KDE¶
KDE is a GUI framework built on Qt.
KWin is the main KDE window manager for X11.
Libc¶
A libc is a standard library of C routines.
Libc implementations:
- Glibc
- https://en.wikipedia.org/wiki/C_standard_library#BSD_libc
- https://en.wikipedia.org/wiki/UClibc
- https://en.wikipedia.org/wiki/Bionic_(software)
Glibc¶
Glibc is the GNU C Library (Libc).
Many Linux packages and the GNU/Linux kernel build from Glibc.
Libcloud¶
Apache libcloud is a Python library which abstracts and unifies a large number of Cloud APIs for Compute Resources, Object Storage, Load Balancing, and DNS.
Salt salt cloud depends upon libcloud.
Libvirt¶
Libvirt is a system for platform virtualization with various Linux hypervisors.
- KVM/QEMU
- Xen
- LXC
- OpenVZ
- VirtualBox
Linux¶
GNU/Linux is a free and open source operating system kernel written in C.
uname -a; echo "Linux"
uname -o; echo "GNU/Linux"
A Linux Distribution is a collection of Packages compiled to work with a GNU/Linux kernel and a Libc.
LLVM¶
LLVM “Low Level Virtual Machine” is a reusable compiler infrastructure with frontends for many languages.
Make¶
GNU Make is a classic, ubiquitous software build tool designed for file-based source code compilation.
Bash, Python, and the GNU/Linux kernel are all built with Make.
Make build task chains are represented in a Makefile
.
Pros
- Simple, easy to read syntax
- Designed to build files on disk
- Nesting:
make -C <path> <taskname>
- Variable Syntax:
$(VARIABLE_NAME)
- Bash completion:
make <tab>
- Python: Parseable with disutils.text_file Text File
- Logging: command names and values to stdout
Cons
- Platform Portability: make is not installed everywhere
- Global Variables: Parametrization with shell scripts
MessagePack¶
MessagePack is a data interchange format with implementations in many languages.
Node.js¶
Node.js is a framework for JavaScript applications written in C, C++, and JavaScript.
OS X¶
OS X is a UNIX operating system based upon the Mach kernel from NeXTSTEP, which was partially derived from NetBSD and FreeBSD.
OS X GUI support is built from XFree86/X.org X11.
OS X maintains forks of many POSIX BSD and GNU tools like bash
,
readlink
, and find
.
Homebrew installs and maintains packages for OS X.
uname; echo "Darwin"
Packer¶
Packer generates machine images for multiple platforms, clouds, and hypervisors from a parameterizable template.
- Packer Artifact
- Build products: machine image and manifest
- Packer Template
- JSON build definitions with optional variables and templating
- Packer Build
- Task defined by a JSON file containing build steps which produce a machine image
- Packer Builder
Packer components which produce machine images for one of many platforms:
- VirtualBox
- Docker
- OpenStack
- GCE
- EC2
- VMware
- QEMU (KVM, Xen)
- http://www.packer.io/docs/templates/builders.html
- Packer Provisioner
Packer components for provisioning machine images at build time
- Shell scripts
- File uploads
- ansible
- chef
- solo
- puppet
- salt
- Packer Post-Processor
- Packer components for compressing and uploading built machine images
Perl¶
Perl is a dynamically typed, C-based third-generation programming language.
Many of the Debian system management tools are or were originally written in Perl.
Python¶
Python is a dynamically-typed, C-based third-generation programming language.
As a multi-paradigm language with support for functional and object-oriented code, Python is often utilized for system administration and scientific software development.
Many of the RedHat system management tools (such as Yum) are written in Python. Gentoo Portage is written in Python.
IPython, Pip, Conda, Sphinx, Docutils, Mercurial, Libcloud, Salt, Tox, Virtualenv, and Virtualenvwrapper are all written in Python.
The Python community is generously supported by a number of sponsors and the Python Infrastructure Team:
- https://www.python.org/psf/sponsorship/
- https://www.python.org/psf/members/#sponsor-members
- http://psf-salt.readthedocs.org/en/latest/overview/
CPython¶
CPython is the reference Python language implementation written in C.
CPython can interface with other C libraries through a number of interfaces:
Cython¶
Cython is a superset of CPython which adds static type definitions; making CPython code faster, in many cases.
NumPy¶
NumPy is a library of array-based mathematical functions implemented in C and Python.
- http://nbviewer.ipython.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-2-Numpy.ipynb
- https://scipy-lectures.github.io/intro/numpy/index.html
- https://scipy-lectures.github.io/advanced/advanced_numpy/index.html
NumPy and other languages:
SciPy¶
SciPy is a set of science and engineering libraries for Python, primarily written in C.
- http://nbviewer.ipython.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-3-Scipy.ipynb
- https://scipy-lectures.github.io/intro/scipy.html
The SciPy Stack specification includes the SciPy package and its dependencies.
PyPy¶
PyPy is a JIT LLVM compiler for Python code written in RPython – a restricted subset of CPython syntax – which compiles to C, and is often faster than CPython for many types of purposes.
NumPyPy¶
Python 3¶
Python 3 made a number of incompatible changes, requiring developers to update and review their Python 2 code in order to “port to” Python 3.
Python 2 will be supported in “no-new-features” status for quite some time.
Python 3 Wall of Superpowers tracks which popular packages have been ported to support Python 3: https://python3wos.appspot.com/
There are a number of projects which help bridge the gap between the two language versions:
- https://pypi.python.org/pypi/six
- http://pythonhosted.org/six/
- https://pypi.python.org/pypi/nine
- https://github.com/nandoflorestan/nine/blob/master/nine/__init__.py
- https://pypi.python.org/pypi/future
- http://python-future.org/
The Anaconda Python distribution (Conda) maintains a working set of packages for Python 2.6, 2.7, 3.3, and 3.4: http://docs.continuum.io/anaconda/pkg-docs.html
awesome-python-testing¶
Pyline¶
Pyline is a UNIX command-line tool for line-based processing in Python with regex and output transform features similar to Grep, Sed, and Awk.
Pyline can generate quoted CSV, JSON, HTML, etc.
Pyrpo¶
Pyrpo is a tool for locating and generating reports from Git, Mercurial, Bazaar, and Subversion repositories.
Readline¶
ReStructuredText¶
ReStructuredText (RST, ReST) is a plaintext lightweight markup language commonly used for narrative documentation and Python docstrings.
Sphinx is built on Docutils, which is the primary implementation of ReStructuredText.
Pandoc also supports a form of ReStructuredText.
- ReStructuredText Directive
Actionable blocks of ReStructuredText
.. include:: goals.rst .. contents:: Table of Contents :depth: 3 .. include:: LICENSE
- ReStructuredText Role
RestructuredText role extensions
.. _anchor-name: :ref:`Anchor <anchor-name>`
Ruby¶
Ruby is a dynamically-typed programming language.
Vagrant is written in Ruby.
Salt¶
Salt is an open source configuration management system for managing one or more physical and virtual machines running various operating systems.
- Salt Top File
- Root of a Salt Environment (
top.sls
) - Salt Environment
- Folder of Salt States with a top.sls top file.
- Salt Bootstrap
- Installer for salt master and/or salt minion
- Salt Minion
Daemon process which executes Salt States on the local machine.
Can run as a background daemon. Can retrieve and execute states from a salt master
Can execute local states in a standalone minion setup:
salt-call --local grains.items
- Salt Minion ID
Machine ID value uniquely identifying a minion instance to a Salt Master.
By default the minion ID is set to the FQDN
python -c 'import socket; print(socket.getfqdn())'
The minion ID can be set explicitly in two ways:
/etc/salt/minion.conf:
id: devserver-123.example.org
/etc/salt/minion_id:
$ hostname -f > /etc/salt/minion_id $ cat /etc/salt/minion_id devserver-123.example.org
- Salt Master
Server daemon which compiles pillar data for and executes commands on Salt Minions:
salt '*' grains.items
- Salt SSH
Execute salt commands and states over SSH without a minion process:
salt-ssh '*' grains.items
- Salt Grains
Static system information keys and values
- hostname
- operating system
- ip address
- interfaces
Show grains on the local system:
salt-call --local grains.items
- Salt Modules
Remote execution functions for files, packages, services, commands.
Can be called with salt-call
- Salt States
Graphs of nodes and attributes which are templated and compiled into ordered sequences of system configuration steps.
Naturally stored in
.sls
YAML files parsed bysalt.states.<state>.py
.Salt States files are processed as Jinja templates (by default) they can access system-specific grains and pillar data at compile time.
- Salt Renderers
- Templating engines (by default: Jinja) for processing templated states and configuration files.
- Salt Pillar
Key Value data interface for storing and making available global and host-specific values for minions: values like hostnames, usernames, and keys.
Pillar configuration must be kept separate from states (e.g. users, keys) but works the same way.
In a master/minion configuration, minions do not have access to the whole pillar.
- Salt Cloud
Salt Cloud can provision cloud image, instance, and networking services with various cloud providers (Libcloud):
- Google Compute Engine (GCE) [KVM]
- Amazon EC2 [Xen]
- Rackspace Cloud [KVM]
- OpenStack [https://wiki.openstack.org/wiki/HypervisorSupportMatrix]
- Linux LXC (Cgroups)
- KVM
Sed¶
GNU Sed is a UNIX CLI utility for transforming text.
Note
BSD Sed
Use <Ctrl-V><tab>
for explicit tabs (as \t
does not work)
Use \\\n
or '$'\n
for newlines (as \n
does not work)
sed -E
should be consistent extended regular expressions
between GNU Sed (e.g. Linux) and BSD Sed (FreeBSD, OSX).
OR: brew install gnu-sed
See: https://unix.stackexchange.com/questions/101059/sed-behaves-different-on-freebsd-and-on-linux
See: https://superuser.com/questions/307165/newlines-in-sed-on-mac-os-x
Sphinx¶
Sphinx is a tool for working with ReStructuredText documentation trees and rendering them into HTML, PDF, LaTeX, ePub, and a number of other formats.
Sphinx extends Docutils with a number of useful markup behaviors which are not supported by other ReStructuredText parsers.
Most other ReStructuredText parsers do not support Sphinx directives; so, for example,
GitHub and BitBucket do not support Sphinx but do support ReStructuredText so README.rst containing Sphinx tags renders in plaintext or raises errors.
For example, the index page of this Sphinx documentation set is generated from a file named
index.rst
and referenced bydocs/conf.py
.
- Sphinx Builder
Render Sphinx ReStructuredText into various forms:
- HTML
- LaTeX
- ePub
See: Sphinx Builders
- Sphinx ReStructuredText
- Sphinx extends ReStructuredText with roles and directives which only work with Sphinx.
- Sphinx Directive
Sphinx extensions of Docutils ReStructuredText directives.
Most other ReStructuredText parsers do not support Sphinx directives.
.. toctree:: readme installation usage
See: Sphinx Directives
- Sphinx Role
Sphinx extensions of Docutils ReStructuredText roles
Most other ReStructured
.. _anchor-name: :ref:`Anchor <anchor-name>`
Tox¶
Tox is a build automation tool designed to build and test Python projects with multiple language versions and environments in separate virtualenvs.
Run the py27 environment:
tox -v -e py27
tox --help
Ubuntu¶
Vagrant¶
Vagrant is a tool for creating and managing virtual machine instances with CPU, RAM, Storage, and Networking.
- Vagrant:
- provides helpful commandline porcelain on top of
VirtualBox
VboxManage
- installs Vagrant Boxes
- provides helpful commandline porcelain on top of
VirtualBox
vagrant help
vagrant status
vagrant init ubuntu/trusty64
vagrant up
vagrant ssh
$EDITOR Vagrantfile
vagrant provision
vagrant halt
vagrant destroy
- Vagrantfile
Vagrant script defining a team of one or more virtual machines and networks.
Create a Vagrantfile:
vagrant init [basebox] cat Vagrantfile
Start virtual machines and networks defined in the Vagrantfile:
vagrant status vagrant up
- Vagrant Box
Vagrant base machine virtual machine image.
There are many baseboxes for various operating systems.
Essentially a virtual disk plus CPU, RAM, Storage, and Networking metadata.
Locally-stored and cached vagrant boxes can be listed with:
vagrant help box vagrant box list
A running vagrant environment can be packaged into a new box with:
vagrant package
Packer generates VirtualBox Vagrant Boxes with a Post-Processor.
- Vagrant Cloud
Vagrant-hosted public Vagrant Box storage.
Install a box from Vagrant cloud:
vagrant init ubuntu/trusty64 vagrant up vagrant ssh
- Vagrant Provider
A driver for running Vagrant Boxes with a hypervisor or in a cloud.
The Vagrant VirtualBox Provider is well-supported.
With Plugins: https://github.com/mitchellh/vagrant/wiki/Available-Vagrant-Plugins
See also: Libcloud.
- Vagrant Provisioner
Set of hooks to install and run shell scripts and configuration managment tools over
vagrant ssh
.Vagrant up runs
vagrant provision
on first invocation ofvagrant up
.vagrant provision
Note
Vagrant configures a default NFS share mounted at /vagrant
.
Note
Vagrant adds a default NAT Adapter as eth0; presumably for
DNS, the default route, and to ensure vagrant ssh
connectivity.
Vim¶
Vimium¶
Vimperator¶
Wasavi¶
VirtualBox¶
Oracle VirtualBox is a platform virtualization package for running one or more guest VMs (virtual machines) within a host system.
VirtualBox:
- runs on many platforms: Linux, OSX, Windows
- has support for full platform NX/AMD-v virtualization
- requires matching kernel modules
Vagrant scripts VirtualBox.
Virtualenv¶
Virtualenv is a tool for creating reproducible Python environments.
Virtualenv sets the shell environment variable $VIRTUAL_ENV
when active.
Virtualenv installs a copy of Python, Setuptools, and Pip when a new virtualenv is created.
A virtualenv is activated by source
-ing ${VIRTUAL_ENV}/bin/activate
.
Paths within a virtualenv are more-or-less FHS standard paths, which makes virtualenv structure very useful for building chroot and container overlays.
A standard virtual environment:
bin/ # pip, easy_install, console_scripts
bin/activate # source bin/activate to work on a virtualenv
include/ # (symlinks to) dev headers (python-dev/python-devel)
lib/ # libraries
lib/python2.7/distutils/
lib/python2.7/site-packages/ # pip and easy_installed packages
local/ # symlinks to bin, include, and lib
src/ # editable requirements (source repositories)
# also useful
etc/ # configuration
var/log # logs
var/run # sockets, PID files
tmp/ # mkstemp temporary files with permission bits
srv/ # local data
Virtualenvwrapper wraps virtualenv.
echo $PATH; echo $VIRTUAL_ENV
python -m site; pip list
virtualenv example # mkvirtualenv example
source ./example/bin/activate # workon example
echo $PATH; echo $VIRTUAL_ENV
python -m site; pip list
ls -altr $VIRTUAL_ENV/lib/python*/site-packages/** # lssitepackages -altr
Note
Venv extends Virtualenv and Virtualenvwrapper.
Note
Python 3.3+ now also contain a script called venv, which performs the same functions and works similarly to virtualenv: https://docs.python.org/3/library/venv.html.
Virtualenvwrapper¶
Virtualenvwrapper is a tool which extends virtualenvwrapper.
Virtualenvwrapper provides a number of
useful shell commands and python functions
for working with and within virtualenvs,
as well as project event scripts (e.g. postactivate
, postmkvirtualenv
)
and two filesystem configuration variables
useful for structuring
development projects of any language within virtualenvs:
$PROJECT_HOME
and $WORKON_HOME
.
Virtualenvwrapper is sourced into the shell:
# pip install --user --upgrade virtualenvwrapper
source ~/.local/bin/virtualenvwrapper.sh
# sudo apt-get install virtualenvwrapper
source /etc/bash_completion.d/virtualenvwrapper
Note
Venv extends Virtualenv and Virtualenvwrapper.
echo $PROJECT_HOME; echo ~/workspace # venv: ~/wrk
cd $PROJECT_HOME # venv: cdp; cdph
echo $WORKON_HOME; echo ~/.virtualenvs # venv: ~/wrk/.ve
cd $WORKON_HOME # venv: cdwh; cdwrk
mkvirtualenv example
workon example # venv: we example
cdvirtualenv; cd $VIRTUAL_ENV # venv: cdv
echo $VIRTUAL_ENV; echo ~/.virtualenvs/example # venv: ~/wrk/.ve/example
mkdir src ; cd src/ # venv: cds; cd $_SRC
pip install -e git+https://github.com/westurner/dotfiles#egg=dotfiles
cd src/dotfiles; cd $VIRTUAL_ENV/src/dotfiles # venv: cdw; cds dotfiles
head README.rst
# venv: cdpylib
cdsitepackages # venv: cdpysite
lssitepackages
deactivate
rmvirtualenv example
lsvirtualenvs; ls -d $WORKON_HOME # venv: lsve; lsve 'ls -d'
Wayland¶
Wayland is a display server protocol for GUI window management.
Wayland is an alternative to X11 servers like XFree86 and X.org.
The reference Wayland implementation, Weston, is written in C.
YAML¶
YAML (“YAML Ain’t Markup Language”) is a concise data serialization format.
Most Salt states and pillar data are written in YAML. Here’s an
example top.sls
file:
base:
'*':
- openssh
'*-webserver':
- webserver
'*-workstation':
- gnome
- i3
ZSH¶
X11¶
X Window System (X, X11) is a display server protocol for window management (drawing windows on the screen).
Most UNIX and Linux systems utilize XFree86 or the newer X.org X11 window managers.
Gnome, KDE, I3wm, OS X, and Compiz build upon X11.