Thread by @alexpotato

So wanted to do a thread on using #bash on the #linux #cli .

Bash often gets ignored in today's cloud centric world but there is a lot of cool stuff you can do just with basic commands, the switches on those commands and piping things together.

To the command line!

So I'm going to start out with some of the more basic commands and some switches that people aren't familiar with and then rapidly get more advanced in both usage and stringing commands together.

Never done this on Twitter before so should be exciting!

Let's start with: ls

# show files in a single column
ls -1

# show files with detail
ls -l

# show files in reverse time order
ls -ltr

# show all hidden files
ls -a

# show files with human readable size and sorted by size
ls -lSh

Up next: cat

# dump all the contents of a file
cat file.txt

# dump with line numbers (useful later)
cat -n

Up next: head and tail

# get first 10 lines of file
head -10 a.txt

# last 10 lines
tail -10 a.txt

# first line of the last 10 lines
tail -10 a.txt | head -1

# keep tailing a file as it updates
tail -f a.txt

Up next: tr

# Switches our characters e.g. swap pipe for newline
tr "|" "\n"

# swap ASCII char \001 for \n, great for people in finance who work with FIX protocol
tr "\001" "\n"

# if there are multiple of char, replace with just one, useful for ps output
tr -s " "

Up next: grep

# find hello in a.txt
grep hello a.txt

# show 2 lines before and 3 lines after match
grep -B 2 -A 3 hello a.txt

# grep for strings in a file
grep -f account.txt a.txt

# just the names of file with match
grep -l hello *.txt

# names and match
grep -H hello *.txt

grep pt2

# recursively search through a dir tree
grep -r hello

# get only the FIRST match in a file
grep -m 1 hello *.txt

# grep and return true if found, false if not. Useful for && commands based on a grep
grep -q hello.txt

# grep output of another command
ls | grep hello

Up next: cut

# from a csv, get all values in the 2nd col
cat a.txt | cut -d"," -f2

# get cols 3-7
cat a.txt | cut -d"," -f3-7

# cols 3,9,11 for pipe delimited
cat a.txt | cut -d"|" -f3,9,11

Up next: sort

# sort output of another command
cat a.txt | sort

# sort using number sort, useful for du -k output
cat a.txt | sort -n

# assuming a pipe delimited file, sort by the 2nd col
cat a.txt | sort -t"|" -k2,2

# sort and get unique values
cat a.txt | sort -u

Up next: uniq

# get uniq values from a file
cat a.txt | sort | uniq

# get uniq values AND how often they appear
cat a.txt | sort | uniq -c

Up next: comm

This is THE single most useful command I wish I had known about 10 years ago.

Takes:
- two sorted lists
- compares them
- allows you to see what's in A but not B
- B but not A
- In both

# e.g. show me only what is in file A but not B
comm -23 a.txt b.txt

# only in B
comm -13 a.txt b.txt

# what values are in both
comm -12 a.txt b.txt

SUPER handy for when there is a network outage and you want to see what was sent from source but didn't get to destination

Up next: find

Has like a million switches so go read the man page. And google "linux find examples" otherwise you will get lost.

# find anything with .pdf in lowercase in the name
find . -iname "*.pdf"

# find file created in the last 30 mins
find . -type f -cmin -30

# find files created in the last 40 days
find . -type f -ctime -40

# find files modified more than 50 days ago
find . -type f -mtime +50

# find files created less than 20 mins and do md5sum on them
# {} below is the found filename
find . -cmin -20 -exec md5sum {}\;

Examples:

# Find multiple files with the same name and see if they are the same contents
find . -iname "app.conf" -exec md5sum {} \; | sort | uniq -c

If you get multiple md5sums back, then you have files with different contents

# Show the fields of a csv with numbers so we know what values to pass to cut
cat a.csv | head -1 | tr "," "\n" | cat -n

Will list all of the fields with their order number so you can use cut more effectively

# grep both gzipped and normal recent logs from multiple apps &s ort by the time (2nd field and pipe delimited logs)

find . -cmin -20 -exec zgrep localhost {} \; | sort -t"|" -k2,2

Above is great when you are trying to track a value (e.g. localhost) across multiple app logs

# fintech & want to comp the tag 17 (exec id) from 2 different logs

# 1st get the unique tag 17s from each log
grep FIX a.log | tr "\001" "\n" | grep "^17=" | sort -u > a.txt

# do the same for b.log
# then do a comm to see which exec ids are not in b log:
comm -23 a.txt b.txt