Make a big file into many small files (and back again)

Adapted from this slashdot thread

Create many little files called "chunks00000" etc. from bigfile that are all just under 10MB in size: split -a 5 -b 10MB -d bigfile chunks

Put the chunks back together: cat * > newbigfile

Do the same, except with compression: tar -zc bigfile | split -a 5 -b 10MB -d - chunks

Put the compressed chunks back together: cat chunks000* | tar -zxO > newbigfile

Funny UNIX tricks from Slashdot

There was a recent story on slashdot about useless (or useful) things one can do in UNIX. Being a command line junkie, I read through virtually every comment (all 2300+ of them) to learn some new tricks. Here are some of the better ones:

Bash History

• history -c # clear history (good for preserving privacy/passwords, or check out the more precise -d option)
  
• In vi command mode, type /query and hit Enter to search history, n to keep searching backwards, N to search forwards

vimdiff

• vimdiff original_file patched_file
  
• unified format: open original file, then :vertical diffpatch path/to/diff

Encryption

• openssl aes-256-cbc -a -e -salt -in INPUT_FILENAME -out OUTPUT_FILENAME # encrypt
• openssl aes-256-cbc -a -d -salt -in INPUT_FILENAME -out OUTPUT_FILENAME # decrypt
• echo Oe lbh pna vzcyrzrag UK tbireazrag fgnaqneq rapelcgvba jvgu ge | tr a-z n-za-m # Rot 13 encrypt/decrypt
  

Others:

• sleep 8h; cat /dev/urandom > /dev/dsp # alarm clock
  
• eject -T # close cd tray if open, open if closed (useful to find out which physical machine you are logged into)
• sl # punish users who accidentally type 'sl' instead of 'ls'
  
• eposd && say 'hello' # make the computer talk
  
• :(){ :|:& };: # forkbomb (space required between { and :) (protect against this with ulimit -u)
  
• for I in $(seq 1 100) ; do echo $I; sleep .25; done | dialog --gauge "PIZZA" 6 50 100 # Pizza timer via dialog

wmctrl and friends

wmctrl seems like an awesome utility. I first read about it in Kyle Rankin's Linux Journal column here. The wmctrl project page also has links to a bunch of other desktop-automation and related utilities. This is all going in the "to learn when I have some spare time" file along with screen.

Common Bash Tasks

I don't use Bash for much scripting. Usually, if I have to open up a file to write a script, it will be done in Python. However, the less I have to do that for mundane tasks that can be accomplished on the command line with Bash, the better. This entry may not have much in it now, but every time I do something cool/useful/time-saving in Bash from now on, it'll go on here.

Common Tasks

• Rename *.foo files to *.bar files: for i in *.foo; do mv $i ${i%.foo}.bar; done (BashFAQ)
• Rename foo.* files to bar.* files: for i in foo.*; do mv $i bar.${i#foo.}; done (also check out the perl regex-based rename Linux utility)
  
• Do something to multiple arguments: for i in arg1 arg2 arg3; do echo $i; done
  
• Print 0 through 9 on separate lines: for i in {0..9}; do echo $i; done
• Flatten output onto one line: <Multi-line output> | xargs (10 habits)
  

Links

• GNU Bash Reference Manual
• Bash Manpage
• BashFaq Bash FAQs and answers
• Bash Pitfalls common mistakes
  
• Advanced Bash Scripting Guide Everything you would want to know about Bash scripting
• Bash Guide for Beginners A beginner's guide to Bash, some shell examples
• 10 Good UNIX Habits Advanced tips on the command line

Finding and Killing a Process in GNOME

There are a few different ways to find and kill a process in Linux. Using the GNOME interface, you can call up the System Monitor (similar to the one on Windows XP) under System->Administration->System Monitor. Under the Processes tab, right clicking on a process will give you a variety of options: stop/end/kill/continue the process, change the priority (nice), display a memory map (pmap) and display open files (lsof -p <procID>).

Embedding the system monitor in a desktop panel is useful for constantly monitoring the system and quickly determining if and where there is a problem. Do this with Right Click on Panel->Add to Panel...->System & Hardware->System Monitor->Add. The display provides graphs of Processor, Memory, Network, Swap Space, Hard disk and Load activity (not quite sure what load is, perhaps some metric computed from all the others? I don't use it, personally).

On the command line, list processes with the ps command or ps -e for all users' processes. Pipe that through grep (i.e., ps -e | grep firefox) to find a process ID of the target process (if you know the exact name of the program, you can also use pidof firefox-bin). To kill a process, use the kill <procID> command (with the -9 flag if you want to kill it unconditionally). Alternatively, if the process uses a gui, you can use xkill to merely click on the gui to kill the corresponding process. There is also killall if you want to kill by process table name (i.e. killall firefox-bin).

pgrep and pkill are two very useful utilities, especially for dealing with processes with long command line invocations. pgrep -fl -u danny "ruby.*asdf" will print the process id and name of every process owned by danny whose command line invocation matches the passed in regular expression. The -f argument tells it to match against the full command line invocation instead of just the process name. pkill -fu dannyc "ruby.*asdf" will kill those same processes (for the life of me, I can't seem to coax similar behavior out of killall).

The top command provides an integrated way of finding and killing a process on the command line. If the process-to-be-killed is using a lot of system resources, it will appear towards the top of the display. Hit k and then the process ID to kill the process. Top comes with a lot of other functionality -- hit h for help.

UPDATE: I recommend using htop instead of top, which is top with less suck. It has lots of good features like color, mouse support, a more responsive interface, a tree-based process view, etc.

whois

The WHOIS database is a nice way to determine contact information for a web site. Query the site with whois <site> It shows the name and address of the registrant, through who he/she registered the site, a technical and administrative contact, and the domain servers corresponding to the site. Use it along with dig, host, ping, etc. (and if you're feeling naughty, nmap) to get information about a site.

Links

• Whois (Wikipedia entry)
• Whois RFC
  

Linux Command Line Odds and Ends

Here are some useful Odds and Ends... most are related to command line stuff, some not; whatever, enjoy. Most of these came from either scouring the web, or Learning the Bash Shell or Learning Red Hat Enterprise Linux and Fedora.

Job Control

• kill %<PID> kill a process
• kill -QUIT %<PID> kill a process, a bit stronger
• kill -KILL %<PID> unconditionally kill a process
• fg bring a background job into the foreground
• jobs list jobs running
• ps process information

File permissions (owner, group, others)

• 0 ---
• 1 --x
• 2 -w-
• 3 -wx
• 4 r--
• 5 r-x
• 6 rw-
• 7 rwx

Globbing

• * matches zero or more characters
• ? matches any one character
• [abc...] matches any of the characters specified
• [a-z] matches any character in the specified range
• [!abc...] matches any character other than those specified
• [!a-z] matches any character not in the specified range
• ~ home directory of current user
• ~userid home directory of a user
• ~+ current working directory
• ~- previous working directory

Quotes

• 'xxx' interprereted literally, variables not substituted
• "xxx" interprereted literally, variables ARE substituted
• `xxx` output of xxx command replaces it

Command line special characters

• # comment
• ; command seperator
• & run in background
• \ command continued on next line
• | pipe

Input/Output Redirectors

• prog > file stdout to file
• prog 2> file stderr to file
• prog >> file concatenates stdout to file
• prog 2>> file concatenates stderr to file
• prog > file 2>&1 stdout and stderr to file
• prog >> file 2>&1 concatenates stdout and stderr to file
• prog < file stdin from file
• prog << text reads stdin until a line matching text is found, then EOF posted ("here document")
• prog | prog2 pipe stdout
• prog 2>&1 | prog2 pipe stdout and stderr
  

Command Line Movement

• Ctrl+Shift+N open new console window
• Crtl+Alt+F[1-7] go to virtual console 1-6 or X(7)
• Ctrl+Alt+Backspace stop X and go to console
• Alt+B Back one word
• Alt+F Forward one word
• Ctrl+A Beginning of line
• Ctrl+E End of line
• Alt+D Delete word |------------->X
• Ctrl+D Delete char
• Ctrl+K Delete |------------->X
• Ctrl+U Delete X<-----------|
• Ctrl+L Clear screen
• Ctrl+Y UNDO
• ESC+. Insert last word of previous command
• TAB Possible completions

IRC

• /server join a server
• /join join a channel
• /quit quit the server
• /close close the current screen
• /part leave the current channel
• /partall leave all channels
• /msg msg a user with a new window
• /notice msg a user without a new window
• /query force a window open to msg a user
• /chat DCC with a user
• /dns dns lookup for a user
• /ping ping a user
• /me *** does something
• /whois query whois for a user

Ctrl Keys

• Ctrl+C intr: stop current command
• Ctrl+D eof: end of input
• Ctrl+\ quit: stop current command (if Ctrl+C doesn't work)
• Ctrl+S stop: halt output to screen
• Ctrl+Q resume output to screen
• Ctrl+Z suspend current command (works well with bg, fg and jobs)
  

Escape Sequences

• \a alert (bell)
• \b backspace
• \c omit final newline
• \E escape character
• \f formfeed
• \n newline
• \r return
• \t tab
• \v vertical tab
• \xxx ASCII in octal
• \\ backslash

Tar: tarball options

• -c create
• -r append
• -t list contents
• -x extract
• -a append files
• -v verbose
• -z zip/unzip
• -f use filename
• Oft-used:
• tar -cf foo.tar foo; gzip foo.tar
• gunzip bar.tar.gz; tar -xvf bar.tar