DevOps · Manfred Touron

6 pages about "DevOps"

Cheat Sheet

SSH / Mosh

More SSH awesomeness on https://github.com/moul/awesome-ssh

SSH or mosh into a host and attach or create a named tmux session

ssh SERVER -t "tmux attach -t mysession || tmux new-session -s mysession || /bin/sh"

and with mosh

mosh SERVER -- /bin/sh -c "tmux attach -t mysession || tmux new-session -s mysession || /bin/sh"

replaces SERVER with your SSH target hostname.

Unsafe SSH or Mosh connection (temporary servers: CI)

ssh -o ControlMaster=no -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no SERVER

with Mosh:

mosh -ssh="ssh -o ControlMaster=no -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" -- SERVER

Non-interactive SSH with password

expect -c "set timeout -1; spawn ssh SERVER; match_max 100000; expect *password:*; send -- PASSWORD\r; interact;"

Networking

Measure network performances and stability with iperf

Install:

Install on mac with brew install iperf
Install on Linux with sudo apt install iperf

TCP test:

Run on the server: iperf -s -p 6666
Run on the client: iperf -c ${SERVER} -p 6666

UDP test:

Run on the server: iperf -s -p 6666 -u
Run on the client: iperf -c ${SERVER} -p 6666 -u -b 900m

traceroute & ping with mytraceroute (mtr)

Install on Linux: sudo apt install mtr
Usage: mtr ${IP}

Example:

$ mtr google.com
                                   My traceroute  [v0.86]
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                   Packets               Pings
 Host                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. xxx-xxx-xxx-xxx.rev.poneytelecom.eu           0.0%     5    0.4   0.6   0.4   0.7   0.0
 2. 195.154.2.6                                   0.0%     4    0.9   0.9   0.8   1.0   0.0
 3. 209.85.149.12                                 0.0%     4    1.0   1.0   1.0   1.0   0.0
 4. 108.170.245.1                                 0.0%     4    2.4   2.2   2.0   2.4   0.0
 5. 66.249.95.247                                 0.0%     4    1.1   1.1   1.1   1.1   0.0
 6. par21s12-in-f14.1e100.net                     0.0%     4    1.1   1.1   1.1   1.2   0.0

Find big files & Cleanup space

Get aggregated sizes of sub-dirs and files

du -hs .??* *

Find big files

find / -xdev -type f -size +100M -ls 2>/dev/null

Cleanup docker

docker system prune

Git / GitHub

Git / GitHub essential tools

Install “hub” (https://hub.github.com/)
Install “refined GitHub” browser extension (https://github.com/sindresorhus/refined-github)

Do rebase instead of merge when pulling

git config --global pull.rebase true

GitHub links

Update the following variables when appropriate:

USER a GitHub user or organization, i.e.: moul
Pull-Requests
- In any repo of an user/organization: https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+archived%3Afalse+user%3AUSER
- Opened by an user: https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+archived%3Afalse+author%3AUSER
- Assignee to an user: https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+archived%3Afalse+assignee%3AUSER
- Mentioning an user/a team: https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+archived%3Afalse+mentions%3AUSER
- Requesting review from an user/a team: https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+archived%3Afalse+review-requested%3AUSER

Close a pull request from command line with “hub”

Install “hub”
Type hub api -X PATCH /repos/$USER/$REPO/pulls/$ID -F state=closed

List projects in JSON

Fetch: hub api --paginate user/repos | jq . > github.json
Only “moul’s” projects: cat github.json | jq '.[] | select(.owner.login == "moul") > moul.json
Remove some fields to have a smaller .json file: cat moul.json | jq 'with_entries(select(.key|test("_url")|not)) | del(.owner) | del(.permissions) | del(.url) | del(.node_id)'

Or inline

hub api --paginate user/repos | jq '.[] | select(.owner.login == "moul") | with_entries(select(.key|test("_url")|not)) | del(.owner) | del(.permissions) | del(.url) | del(.node_id)'`

Bonus: find and close a pull-request named “Enabled GuardRails” in the current repository

export prid=$(hub pr list | grep "Enable GuardRails" | cut -d '#' -f 2 | awk '{print $1}')
if [ "$prid" != "" ]; then
    project=$(git remote -v | cut -d: -f2 | sed "s/\\.git.*//" | head -1)
    hub api -X PATCH /repos/$project/pulls/$prid -F state=closed
fi

Manipulate JSON

Flatten concatenated JSON files

Sometimes, when you run a command like this: hub api --paginate user/repos, you get multiple JSON outputs in the same file.

jq doesn’t care and you can run any jq command over the aggregated file, however, this .json file won’t be supported by most of the other softwares.

You can “repair” by flattening a concatenated JSON file by doing the following:

cat aggregate.json | sed 's/^]$$/],/;$$s/],/]]/;1s/^/[/' | jq '. | add'

The sed ... command will transfer the standalones arrays into a a valid array of arrays.

The jq . | add will flatten the array of arrays into a big 1 dimension array.

Apply change on multiple files

Replace a string by another in a whole Git repository

I’m using my https://github.com/moul/golang-repo-template project as repo template on GitHub.

The first task I need to do immediately after creating the new repo, is to replace every instance of golang-repo-template in the codebase by lorem-ipsum (the project name).

for file in $(git grep golang-repo-template | cut -d: -f1 | sort -u); do sed -i'' s/golang-repo-template/lorem-ipsum/g $file; done

Terminator on Windows

Install VcXsrv.exe on Windows and configure it to start automatically with the system
Create a startTerminator.sh file with the following content: cd; export DISPLAY=:0; nohup terminator &; sleep 1 and change the permissions chmod +x startTerminator.sh
Create a shortcut C:\Windows\System32\bash.exe {12345678-1234-5678-0123-456789abcdef} /home/moul/startTerminator.sh

Web

Temporary static web server

python -m SimpleHTTPServer

And then open your browser on http://localhost:8000

Generate directory listing

tree -H . --charset=utf-8 > index.html

Proxy to avoid caching

{ nc ${proxy_addr} ${proxy_port} && exec 1>&- ; } < <(
    awk '{ sub(/GET/, "POST"); print; fflush("") }; NR<2{print "Expires: Thu, 01 Dec 2042 16:00:00 GMT"; print "Cache-Control: public"; }
)

Socat

Connect to a TTY over TCP

socat stdio,raw,echo=0,escape=0x11 tcp-connect:${HOST}:${PORT}

TCP to UDP proxy

socat TCP-LISTEN:9887,reuseaddr,fork UDP:127.0.0.1:9887

This TCP proxy can be forwarded over SSH to have access to an UDP port from localhost from a remote host

Media manipulation

Loop a video with ffmpeg

calculate the amount of loops needed using https://dan.hersam.com/tools/time-calc.html
for i in {1..383}; do printf "file '%s'\n" "loop1.mp4" >> list.txt; done
ffmpeg -f concat -i list.txt -c copy output.mp4

See this status

Download YouTube video cover image

the image of one video: youtube-dl --write-thumbnail --skip-download https://www.youtube.com/watch?v=xxxxxxxx
the image of multiple videos: youtube-dl --write-thumbnail --skip-download -a urls.txt with one URL per line in urls.txt

#code (51) #devops (6) #cheatsheet (3)

#code (51)