Serialize Shell Script for Transfer

In March, I explored how to remotely execute a script with terminator. 3 months later, the process becomes maturer and I would like to share it here.

The encode/decode process becomes a standalone python script.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

#!/usr/bin/env python
# encoding: utf-8

# usage: serialize-sh.py ~/loki/env/bash_aliases_shared
import zlib, sys, base64

f = open(sys.argv[1], 'r')
output = ''
for line in f:
    line = line.lstrip()
    if len(line) > 0 and line[0] != "#":
        output = output + line

output = base64.b64encode(output.encode('zlib'))
output = "echo " + output + " | base64 --decode | python -c 'import zlib,sys;sys.stdout.write(zlib.decompress(sys.stdin.read()))' > /tmp/source; source /tmp/source; rm /tmp/source"

sys.stdout.write(output)

It takes a file path as parameter and generate a one-liner for execution.

Then it would be easy to trigger it with tmux shortcut.

1

bind-key -n "M-o" run 'tmux send-keys " $(~/loki/env/bin/serialize-sh.py ~/loki/env/bash_aliases_shared)" Enter'

Now spin up your docker container, Alt-o to source your shortcuts and enjoy.

Execute Common Commands via Tmux

In a previous post, we tried to source our shell aliases and functions in remote environments thru Terminator. One month later, I found still too much keystorkes for running my shortcuts. It gets worse when executing commands on multiple VMs without devops tools. So this time let’s play around Tmux.

The goal is to execute commands from a pre-defined library, on any environment, directly without any loading or sourcing.

The Tmux Key Binding

Paste line below in your tmux.conf.

1

bind-key -n "M-c" run 'bash -ic "ask-and-run"'

This line simply say when you press Alt-C, execute the shell command ask-and-run.

Implemention

Now let’s implement the ask-and-run function in .bash-aliases.

1
2
3
4
5
6
7
8
9
10
11
12
13

# used in tmux, alt c, for quickly triggering pre-defined commands
function ask-and-run() {
  items=(
    "ifdown eth0; ifup eth0"
    "vim /etc/network/interfaces"
  )
  cmd=$(zenity --list --column=Commands --height=500 --width=400 "${items[@]}")
  if [[ "$cmd" != "" ]]; then
    chars=$(echo `echo "$cmd" | wc -c` "/2 -1" | bc)
    final=$(echo $cmd | head -c$chars)
    tmux send-keys "$final" Enter
  fi
}

The 2 lines after zenity --list is for converting output from ls|ls to ls. This is needed in zenity 3.8.0. Simply skip that if your zenity is 3.4.0.

That’s all for the setup! Put your favourite commands inside the array items, reload tmux and enjoy Alt-C.

Shortcuts for Docker Learner

As a newbie of docker, I created some bash functions with auto-completion to help exploring docker. With tips in the previous post, you may use them in both local and remote environment.

The Code

Paste file below in e.g. ~/.bash_aliases_shared.

Updated on 4 Jun, so shortcuts below may be different comparing to the gif above.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

# install docker
alias install-docker="wget -qO- https://get.docker.com/ | sh"

# list all running containers
function dls {
  a=$(sudo docker ps -aq | xargs sudo docker inspect --format="  ")
  echo "$a" | sed -e 's/.\{58\} \//  /' -e 's/ false/(down)/' -e 's/ true/(up)/'
}

# list all running containers (name only)
function dlsn {
  dls | awk '{print $2}' | sed -e 's/(.*//'
}

# view logs of a docker container
function dlog { sudo docker logs $1 $2; }

# go into a container (linux kernel > 3.18)
function dbind { tmux rename-window $1; sudo docker exec -i -t $1 bash; }

# remove a container
function drm { sudo docker kill $1; sudo docker rm $1; }

# create a detached container
# usage: dn [image] [name]
# default: dn ubuntu <random>
function dn {
  image=${1-ubuntu}
  if [[ "$2" != "" ]]; then
    name_opts="--name=$2"
  fi
  id=$(sudo docker run -d -v `pwd`:/host $name_opts $image tail -f /dev/stdout)
  dbind $id
}

# list image
alias di="sudo docker images"

# turn container into image
# usage: dci container1 image1
function dci { msg=${1-nomsg}; docker commit -m "$msg" $1 $2; }

# create a new persistant ubuntu 14 container and bind in it
# usage: dnewu14 ubuntu2
function dnu { dn ubuntu $1; }

# usage: dcp a.txt container1
function dcp { tar -cf - $1 | docker exec -i $2 /bin/tar -C / -xf -; }

# auto completion for dlog/dbind/drm
if type complete > /dev/null; then
  _docker_containers() {
    COMPREPLY=( $( compgen -W "$( dlsn )" -- ${COMP_WORDS[COMP_CWORD]} ) ); return 0
  }
  complete -F _docker_containers dlog
  complete -F _docker_containers dbind
  complete -F _docker_containers drm
fi

Bring Your Shortcuts to Any SSH Sessions

Developers have their own tools to work. May not be the coolest but best suit their workflow. Let’s write a Terminator (v. 0.97) plugin and source those shortcuts into any environment.

The Terminator Plugin

Paste file below in ~/.config/terminator/plugins/source_helpers.py.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

#!/usr/bin/python
"""source_helpers.py - Terminator Plugin to source shared cli shortcuts"""

import os
import gtk
import terminatorlib.plugin as plugin
from terminatorlib.translation import _
from terminatorlib.util import widget_pixbuf

# Every plugin you want Terminator to load *must* be listed in 'AVAILABLE'
AVAILABLE = ['SourceHelper']

class SourceHelper(plugin.MenuItem):
    """Add custom commands to the terminal menu"""
    capabilities = ['terminal_menu']
    dialog_action = gtk.FILE_CHOOSER_ACTION_SAVE
    dialog_buttons = (gtk.STOCK_CANCEL, gtk.RESPONSE_CANCEL,
                      gtk.STOCK_SAVE, gtk.RESPONSE_OK)

    def __init__(self):
        plugin.MenuItem.__init__(self)

    def callback(self, menuitems, menu, terminal):
        """Add our menu items to the menu"""
        item = gtk.MenuItem(_('Terminal SourceHelper'))
        item.connect("activate", self.terminal_source_helpers, terminal)
        menuitems.append(item)

    def terminal_source_helpers(self, _widget, terminal):
        """Handle the source_helpers of terminal"""
        f = open(os.getenv("HOME") + '/loki/env/bash_aliases_shared', 'r')
        output = ''
        for line in f:
            line = line.strip()
            if len(line) > 0 and line[0] != "#":
                output = output + line + " ; "
        output = base64.b64encode(output.encode('zlib'))
        output = "echo " + output + " | base64 --decode | python -c 'import zlib,sys;sys.stdout.write(zlib.decompress(sys.stdin.read()))' > /tmp/source; source /tmp/source; rm /tmp/source; clear"
        terminal.vte.feed_child(" " + output + "\n")

Terminator is not a must, a key binding for tmux will do if you get the idea.

Aliases and Functions

Extract aliases and functions that should be shared into a separated file, e.g. ~/.bash_aliases_shared.

1
2
3
4
5
6
7
8
9

# aliases / functions defined here will be usable in both local and remote machine

# do not store cmd started with whitespace into history
HISTCONTROL=ignoreboth

# better output json
alias json='python -mjson.tool'

# you can add your own stuffs here, even auto-completion setup

Add line below in your ~/.bash_aliases, so they can be used in local as well.

1

source ~/.bash_aliases_shared

Bonus

Terminator usually come in handy when reading local settings on remote environment. One use case is to reset the terminal size. You should encountered that ssh session won’t resize when the terminal window is resized. New area cannot be filled. Simply craft another Terminator plugin to solve this problem. It is much like the previous plugin but we resize the session with stty instead.

1
2
3
4
5
6

# ... plugin instantiation and menu registration ...

def terminal_resize(self, _widget, terminal):
    cols = terminal.vte.get_column_count()
    rows = terminal.vte.get_row_count()
    terminal.vte.feed_child("stty cols " + str(cols) + "; stty rows " + str(rows) + "\n")

Shell Inside VIM

Hacking VIM is one of my favorite activity because I use it almost everyday. Interacting with shell inside VIM give us vim’s editing power, snippets, undo and so many great features. I wrote shellbridge 3 months ago and I mostly use it for redis operations and mass file management with SCM.

shellbridge is a daemon written in javascript, enabling an interactive shell experience inside editors like vim. Inspired by xiki but in a different way.

Build a File Finder for Zsh

I have used zsh for 2+ years and I am really happy with it. Sometime I have to insert path of files, usually I have to type in characters, tab or /, characters, tab or /, etc… I want to have something like Ctrl-P for vim for me to find the path quickly and insert to my current editing command. I also want to do it again and again, like committing multiple files into git with one single command.

A plugin of zaw

Building from scratch is painful so I decided to build as a plugin on top of zaw.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

function zaw-src-fuzzy() {
  # Get the file list by find .
  OLDIFS=$IFS
  IFS=$'\n'
  candidates=($(find .))
  candidates=(${(iou)candidates[@]})
  IFS=$OLDIFS

  # Define what kind of action can be performed on the selected item
  # first: accept-line
  # second: accept-search
  actions=("zaw-callback-execute" "zaw-callback-append-to-buffer")
  act_descriptions=("execute" "append to edit buffer")
}
# Register our plugin
zaw-register-src -n fuzzy zaw-src-fuzzy

# Setup Ctrl-F shortcut to trigger
function fuzzy-start { kill-line; zaw-fuzzy }
zle -N fuzzy-start
bindkey '^F' fuzzy-start

Source in your .zshrc file, press ctrl-f in zsh and enjoy.

Pry Tips

Pry is a powerful tool for debugging ruby codes. It’s idea is brilliantly awesome. It’s source code is easy to read and understand. Pry is already an irreplaceable part of my dev. env.. Below I am going to share a bit how I tuned it, hope that is also interesting to you. This one is quite tricky so before moving on, please try Pry a while if you haven’t used it before.

Focus when Entering Session

Pry allows us to add hooks, block below will be executed every time entering Pry session. Add it into your ~/.pryrc and it will ask Tmux to select the window running Pry, and bring the terminal on top with wmctrl.

1
2
3
4
5
6
7
8

Pry.config.hooks.add_hook(:before_session, :focus) do
  _pane = ENV['TMUX_PANE']
  if _pane
    _wid = `tmux list-window -t main -F '\#{window_index} \#{window_id}' | grep '@#{ _pane[1..-1] }$'`.split(' ')[0]
    `tmux select-window -t main:#{_wid}`
  end
  `wmctrl -a "Main Tmux Session"`
end

As you can see, this is very specific for my env. (assuming a tmux has session named ‘main’ and is running under terminal with specific title). Please feel free to modify it to fit you.

Source Code Editing

In a Pry session, you can dynamically modify source codes with the edit command. Pry allows us to configure which editor to use. Do you remember how I open file in project-based vim using a custom launcher? We can apply it in Pry too.

1
2
3

Pry.config.editor = proc { |file, line|
  "xdg-open 'vim://#{file}:#{line}'"
}

Trigger Code Reload

After modifying a source file, we will want Pry to reload dynamically without restarting our application. The Pry.load method come in handy.

1
2
3
4
5
6
7
8

" call pry to reload the current file
" example: Pry.load "/home/loki/projects/loper/app.rb"
function! ReloadCurrentFileInPry()
  let path = expand("%:p")
  let sendcmd = "send-keys -t main '" . 'Pry.load "' . path . '"' . "' Enter"
  call system("tmux " . sendcmd)
endfunction
nnoremap <leader>pp :call ReloadCurrentFileInPry()<cr>

Now whenever you press <leader>pp in VIM, a Pry.load command will be sent to the current Tmux main session.

Plugins

There are 2 must-have plugins to me. Check their github page for details.

  1. pry-stack_explorer

    Powerful tool to travel inside call stacks. When error occurred, show-stack returns the detailed call stacks. Locate the possible method caused problem then up <#id> to go into the context and debug.

  2. pry-clipboard

    Copying commands into system clipboard, later on we can paste into our source codes.

Simple Rails Helper for Bypassing Action Logic

When modifying a view, we check the HTML output very often. Sometime refreshing browser is expensive because of the complex controller logic like fetching external resources. We can stub data at the very beginning but we still have to test with real data in some point of time. I wrote a simple rails helper, to memorize all instance variables defined before rendering a view. Then it can be reused for rendering next time.

The Helper

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

def iv_cache
  raise "Block is expected" unless block_given?
  return yield if Rails.env != 'development'
  id = "#{self.controller_path}##{self.action_name}"
  $iv_cache.try :delete, id if params.has_key?(:no_iv_cache)
  if $iv_cache && $iv_cache.has_key?(id)
    $iv_cache[id].each { |k, v|
      self.instance_variable_set(k, v)
    }
  else
    yield
    $iv_cache ||= {}
    $iv_cache[id] = self.instance_variables.each_with_object({}) { |key, h|
      h[key.to_s] = self.instance_variable_get(key)
    }
  end
end

The Controller

1
2
3
4
5

def show
  iv_cache {
    @post = Models::Post.get(params[:id])
  }
end

Custom Launcher to Fire Vim

Launching VIM from browser empowered us to build efficient workflow. Last time I introduced how to open a partial with Chrome Native Messaging. Another way is using launcher. rails-footnotes launching editor using launcher by default. We will take gvim as an example.

Let’s design the protocol first. In this setup, I want gvim to be launched when accessing vim:///home/loki/some/file.txt:9.

The Launcher

Setup below script and named vimp.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#!/usr/bin/env bash

# xdg-open "vim:///home/loki/loki/loper/trunk/project.vim:9"

# grab filepath from first parameter
# eg. vim:///home/loki/loki/loper/trunk/project.vim:9
fullpath=$1

# split filepath to filename and line number
arr=(`echo $fullpath | sed -e 's/^vim:\/\/\(.*\):\(.*\)$/\1 \2/'`)

# determine stage from filename
stage=$( echo ${arr[0]} | sed -e "s#$HOME/##" | cut -d'/' -f1,2 )

# if stage found in server list (opened with servername)
if vim --serverlist | grep -qi $stage ; then
  # open file and that line on the remote vim
  launch $stage
  vim --servername $stage --remote-send ":tab drop ${arr[0]}<cr>${arr[1]}G"

else
  # open new gvim session
  gvim ${arr[0]} +${arr[1]}
fi

The Launcher Install Script

Please place file below next to the launcher script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

#!/usr/bin/env bash

set -e

echo "Get vimp executable path"

# Absolute path to this script, e.g. /home/user/bin/foo.sh
SCRIPT=$(readlink -f "$0")
# Absolute path this script is in, thus /home/user/bin
base=$(dirname "$SCRIPT")
vimp_path="$base/vimp"

echo "Generate vimp.desktop and write to /usr/share/applications"

echo "
[Desktop Entry]
Encoding=UTF-8
Name=GVIM
Comment=Edit text files in GVIM session
Exec=$vimp_path %U
Terminal=false
Type=Application
Icon=/usr/src/vim/runtime/vim48x48.xpm
Categories=Application;Utility;TextEditor;
MimeType=text/plain;x-scheme-handler/vim;
StartupNotify=true
StartupWMClass=VIMP
" > ~/vimp.desktop

sudo mv ~/vimp.desktop /usr/share/applications/vimp.desktop

echo "Update Desktop Database"
sudo update-desktop-database

echo "Done!"

After setup, you may execute ./path/to/install/script.sh to register our vim launcher.

Verify

1

xdg-open "vim:///home/loki/.vimrc:9"

Make it Work with rails-footnotes

1
2
3

#...skipped from initializers/rails_footnotes.rb
f.prefix = 'vim://%s:%d'
#...skipped

Locate View Partial Template in 1 Click

Problem

When debugging or revamping web pages, we inspect views and partials inside. When complicated page comes in, locating codes can be time consuming, especially when we are not familiar with the page.

The Idea

Chrome extension for finding out partials rendered.

Click the partial name to hit the exact file.

To implement, we need

  1. an automated way to add extra markups
  2. a chrome extension to visualize partials and listen to user action
  3. a background script to launch the text editor

The Markups

The partial info must come from somewhere. Below I take sinatra and erb as an example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# inside the configure do block (not production)
# override the prepare method of tilt
Tilt::ERBTemplate.class_eval do
  def prepare
    @outvar = options[:outvar] || self.class.default_output_variable
    options[:trim] = '<>' if !(options[:trim] == false) && (options[:trim].nil? || options[:trim] == true)

    unless data.lines.first.start_with? "<!doctype"
      data and data.prepend <<-TAG
        <i class="erb-locator" style="display:none; visibility:hidden">#{file}</i>
      TAG
    end

    @engine = ::ERB.new(data, options[:safe], options[:trim], @outvar)
  end
end unless $production

Now <i> tags should be generated right above any partials rendered.

1
2
3

<i class="erb-locator" style="display:none; visibility:hidden">
  /home/loki/loper/trunk/app/views/prod/main/mock-index.erb
</i>

Hacking ActionView::Renderer#render_partial gives you similar result in rails.

The Chrome Extension

Next we need a chrome extension to visualize those <i> tags. Launch text editor when user clicked on it.

1
2
3
4
5
6
7
8
9
10
11
12
13

// manifest.json

{
  // We have to hardcode a private key here for generating fixed extension ID
  // later our ruby script can be called within this extension
  "key": "...private key...",
  ...
  "permissions": [
    "activeTab",         // permission for getting content from active tab
    "nativeMessaging"    // permission for talking to local scripts
  ],
  ...
}

As you can see, a fixed extension ID and Native Messaging support are required.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

// background.js

// Execute the erb locator script when user clicked on a locator
chrome.runtime.onMessage.addListener(function(request, sender, sendResponse) {
  if (request.erb) {
    console.log("Received", request);
    chrome.runtime.sendNativeMessage('com.loki.erb_locator',
      { erb: request.erb },
      function() {
        console.log("Native Message Sent");
      }
    );
    sendResponse({});
  }
});

// Extension entry point
chrome.browserAction.onClicked.addListener(function(tab) {
  chrome.tabs.executeScript(null, {file:"jquery-1.10.2.min.js"}, function() {
    chrome.tabs.executeScript(null, {file:"inject.js"});
  });
});

We have done 2 things here.

  1. added a listener to trigger the erb locator ruby script
  2. injected inject.js to the active tab when user activating our extension

Finally, inject.js for visualizing locators and listening to user action.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

// inject.js

// the backdrop, we will place our locators on it
var $mask = null;

// Open partial in text editor thru Native Messaging
function openERB(erb) {
  chrome.runtime.sendMessage({erb: erb}, function() {
    console.log("Message Sent: " + erb);
  });
}

// locate and visualize those locator
function buildLinks() {
  // keep track on locators with same position for avoid overlapping
  var count = {};

  $('i.erb-locator').each(function() {
    var $this = $(this);
    var path = $this.text();
    var erbName = path.replace(/.*\/(.*).erb/, "$1");

    var $a = $('<a class="_erb">' + erbName + '</a>').attr({
      "href":     "javascript:void(0)",
      "data-erb": path
    }).css({
      "position":   "absolute",
      "border":     "1px solid #888",
      "padding":    "2px 6px",
      "font-size":  "14px",
      "color":      "#111",
      "background": "white"
    });
    // slightly add offsets to overlapped nodes
    var o = $this.next().offset();
    var k = o.top * 1000 + o.left;
    count[k] = count[k] == undefined ? 0 : count[k] + 1;
    $a.offset({top: o.top + count[k] * 19, left: o.left});
    $mask.append($a);
  });
  $('body').append($mask);
}

// select the backdrop out
$mask = $('#_mask');

// remove it when it is already exist
// user triggered the extension again without selecting any one
if (!!$mask.length) {
  $mask.remove();

} else {
  $mask = $('<div id="_mask">');
  $mask.css({
    "position":         "absolute",
    "top":              "0",
    "left":             "0",
    "right":            "0",
    "bottom":           "0",
    "background-color": "rgba(255, 255, 255, 0.5)",
    "z-index":          "99999"
  });

  buildLinks();
  $mask.on('click', 'a._erb', function() {
    openERB($(this).data('erb'));
    $mask.remove();
    return false;
  });
}

The Backend

Mentioned quite a lot the locator ruby script, below is an example and you may roll out your own.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

#!/usr/bin/env ruby

require 'syslog'
require 'json'

Syslog.open

# start here
Syslog.info "Native Message Received, RUBY_VERSION: #{RUBY_VERSION}"

# read number of bytes from stdin
def readc(n)
  n.times.map { STDIN.getc }.join
end

# read msg from chrome native messaging
def readmsg
  len = readc(4).unpack("i")[0]
  readc len
end

# read filename
msg = readmsg
Syslog.info "msg: #{msg}"
filename = JSON.load(msg)["erb"]

system "vim --servername main --remote-send ':tab drop #{filename}<cr><cr>'"
system "wmctrl -a 'GVIM main'"

To let chrome talk to the script, we have to register a Native Messaging Host. For Linux host, place file below under ~/.config/google-chrome/NativeMessagingHosts.

1
2
3
4
5
6
7
8
9
10

// com.loki.erb_locator.json
{
  "name": "com.loki.erb_locator",
  "description": "ERB Locator",
  "path": "/home/loki/loki/env/bin/erb-locator",
  "type": "stdio",
  "allowed_origins": [
    "chrome-extension://odiihfohmigfneenjpfckobhboicfpgn/"
  ]
}