We’ve got a few new people at work who don’t have any Linux experience, and I was asked to do a quick crash course on some super fundamental logging in / navigating / restarting service stuff so their first on call rotation wouldn’t be quite so stressful. Publishing the overview here in case it is useful for anyone else.

Linux Primer:

Connecting – We use both putty and Cygwin to connect to our Linux hosts via SSH (secure socket shell). Each has its own advantages and disadvantages – try them both and see which you prefer. If you need X redirection (you need the GUI ‘stuff’ to magic itself onto your computer), use Cygwin-X.

Logging In – Our Linux hosts authenticate users via cusoldap.windstream.com, so (assuming you are set up for access to the specific host) you will use your CSO userID and password to log in.

• We often use a jump box – log into the jump box with your ID or using a key exchange. From there, we have key exchanges with our other boxes that allow us to connect without entering credentials again.
• You can set up key exchanges on your own ID too – even from your Windows desktop – and avoid typing passwords.

Once you are logged in, you can start a screen session. Normally, anything you are running is terminated if your SSH session terminates (e.g. if you use Cygwin or Putty to connect to a box from your laptop that is VPN’d into the network & your VPN drops … everything you were doing in the SSH session is terminated.). You can use screen to set up a persistent session – you can reconnect to that session should your SSH connection get interrupted, other people can connect to the session to monitor a long running script, or multiple people can connect to the session and all see the same thing (screen sharing).

To start a new screen session, screen -S SessionName where SessionName is something that identifies the screen session as yours (e.g. LJRPasswordResync was the session I used when resyncing all employee and contractor passwords for OIDM – this includes both my initials and the function I’m running in the session). To see the currently running sessions, use screen –ls

[lisa@server810 ~]# screen -ls

There is a screen on:

8210.LJR        (Detached)

1 Socket in /tmp/screens/S-lisa.

The output contains both a session ID number (green) and a session name (blue) separated by a full stop. You can use either to connect to a screen session (the name is case sensitive!). To reconnect, use screen –x SessionName or screen –x SessionID

To determine if you are currently in a screen session, look at the upper left hand corner of your Putty window. The title will change to include screen when you are in a screen session. Alternately echo the STY environment variable. If you get nothing, it is not a screen session. If you get output, it is the PID and name of your current screen session.

[lisa@server810 ~]# echo $STY
43116.LJR

SUDO – The sudo command lets you execute commands that your ID is not normally privileged to run. There is configuration to sudo (maintained by ITSecurity) that defines what you can run through sudo. If, for example, you are unable to edit a file but are permitted to sudo vim … editing a file using “vi /path/to/file.xtn” will throw an error if you attempt to save changes, but running “sudo vi /path/to/file.xtn” would allow you to save changes to the file.

Substitute user – The command su lets you substitute a uidnumber for yours – this means you become that user.

Combining SUDO and SU – Once we are logged into LX810 with our user ID, we can use sudo su – root to become root without actually knowing the root password. The “space dash space” in the su command means the user’s environment is loaded. If you omit the space dash space, you’ll still be logged in as the root user, but your environment will be left in place.

Generally speaking, allowing sudo to root is a bad idea (i.e. don’t do this even though you’ll see it on a lot of our old servers). This is because root has full access to everything and running the shell as root is insecure and typos can be disastrous.

Navigating – You are in a DOS-like command line interface. The interface is known as a shell – root on LX810 is a bash shell. The default for a CUSO ID is the korn shell (/bin/ksh) – you can change your shell in your LDAP account to /bin/bash (or /bin/csh for the C shell) and subsequent logons will use the new shell. You can try each one and see which you prefer, you can use korn because it is the default from CUSO, or you can use bash because it matches the instructions I write.

From a file system navigation perspective, you will be in the logon user’s home directory. If you aren’t sure where you are in the file system, type pwd and the present working directory will be output.

To see what is in a directory, use ls … there are additional parameters you can pass (in Linux parameters are passed with a dash or two dashes). Adding -a lists *all* files (including the hidden ones, any file where the name starts with a full stop is a hidden file). Adding -l provides a long listing (file owners, sizes, modified dates). Adding -h lists file sizes in human readable format. You can pass each parameter separately (ls –a –l –h) or by concatenating them together (ls –alh)

You can use wc to count the number of lines either in a file (wc –l /path/to/file.xtn) or the output of ls (ls –al | wc –l) – this is useful on our sendmail servers when you have received a queue length alert and done something to clear out some of the queue. In sendmail particularly, there are two files for each message so you need to divide the line count by 2.

To change to a different directory, use cd – e.g. cd /etc/mail will change the working directory to /etc/mail.

To delete a file, use rm /path/to/file.xtn – this is the safe way to run it, it will prompt for confirmation for each file being deleted. You can use wildcards (rm /path/to/files*) to delete multiple files. You can add a -f parameter to not be prompted – which is more dangerous as you may have typed the wrong thing and it’ll be deleted without prompting. You can add a –r parameter for recursive (get rid of everything under a path). Not too dangerous as long as you have the prompt coming up – but if you use –r in conjunction with –f (rm –rf) … you can do a lot of damage. Absolute worst case would be recursive force delete from / … which would mean every file on disk goes away. Don’t do that J

If you are not sure where a file you need is located, you can use either find or locate. The locate command is not always installed, so you would need to use the find command. Locate uses an index database – so it’s quicker, but it doesn’t know about files created/deleted since the index was updated.

To use locate, use locate -i filename where filename is some part of the filename. The -i performs a case insensitive search – if you know the proper casing, you do not need to include this parameter.

To use find, you need to indicate the root of the search (if you have no clue, use ‘/’ which is the top level directory) as well as the exact file name that you want (not a substring of the file name like locate will let you do). Finding a file named audit.log that is somewhere on the disk would be find / -name audit.log

Customizing shell environment – You can customize your shell environment. The system-wide shell environment settings are in /etc and are specific to the shell. For a bash shell, it is /etc/bashrc

Individual user settings are in a hidden file within their home directory. For the bash shell, the user specific settings are in $HOME/.bashrc ($HOME is a variable for the current logon user’s home directory).

For a shared account, adding things to $HOME/.bashrc isn’t the best idea – your preferred settings can differ from someone else’s preferences. We make our own rc file in $HOME for the shared account (I actually set my .bashrc as world-readable and linked the shared ID $HOME/.ljlrc to my personal .bashrc file so I only have to remember to edit one file). You can load your personal preferences using source $HOME/.yourrc or you can load someone else’s preferences by sourcing their file in the shared account’s home directory (source $HOME/.ljlrc will load in mine).

Service Control – Most of our Linux systems still use systemd (init.d scripts) to start and stop services. You can find the scripts in /etc/init.d – these are readable text scripts. All scripts will have a start and stop command, and many have restart and status as additional commands. To control a service, you can use service servicename command, /sbin/service servicename command or /etc/init.d/servicename command – same thing either way. If you are controlling the service through sudo, though, you need to use the technique that is permitted to your UID in the sudo configuration.

If you use a command that isn’t implemented in the script, you will get usage information. You can use a semicolon to chain commands (like the & operator in DOS) – so /etc/init.d/sendmail restart is the same thing as running /etc/init.d/sendmail stop;/etc/init.d/sendmail start

Process utilization – To see what the processor and memory utilization is like on a box (as well as which processes are causing this utilization), use top. When top has launched, the first few lines give you the overall usage. The load average (blue below) tells you the load during the last one, five, and fifteen minutes – 1.00 is 100% on a single core system, 2.00 is 100% on a two core system, etc. Over the 100% number for a system means stuff got queued waiting for CPU cycles to become available.

The current CPU utilization (green below) breaks out usage by user tasks, system tasks, nice’d processes (generally nothing here), idle, io wait, hardware irq, software irq.

The memory usage (red below) shows used and free memory.

top – 13:58:30 up 486 days,  2:16,  9 users,  load average: 0.34, 0.24, 0.25

Tasks: 162 total,   1 running, 161 sleeping,   0 stopped,   0 zombie

Cpu(s):  0.4% us,  0.1% sy,  0.0% ni, 99.5% id,  0.0% wa,  0.0% hi,  0.0% si

Mem:   4147208k total,  2107876k used,  2039332k free,    62372k buffers

Swap:  2064376k total,     1352k used,  2063024k free,  1167652k cached

 

The process list can be sorted by whatever you need – if the box is CPU-bound, type an upper case C to sort by CPU usage. If it is memory bound, type an upper case M to sort by memory usage.

PID USER      PR  NI %CPU    TIME+  %MEM  VIRT  RES  SHR S COMMAND

23190 root      15   0    1   5:43.81 14.9  608m 605m 2872 S perl

14225 root      16   0    0   7:14.20  1.7  170m  69m  60m S cvd

14226 root      16   0    0   1:30.32  1.4  147m  57m  50m S EvMgrC

4585 root      16   0    0 212:01.99  1.1  230m  43m 6368 S dsm_om_connsvc3

4003 root      16   0    0   2729:44  0.6  171m  24m 3364 S dsm_sa_datamgr3

24552 root      16   0   13   0:36.16  0.3 17804  12m 2900 S perl

 

The first column shows the PID (process ID). Some commands as listed in top are obvious what they actually are (httpd is the apache web server, for instance) and others aren’t (perl, above, doesn’t really tell us *what* is using the CPU). To determine what the PID actually is, use ps –efww | grep PID#

[lisa@server810 Sendmail-CheckQSize]# ps -efww | grep 23190

root     23190 23187  0 01:23 ?        00:05:44 /usr/bin/perl /home/NDSSupport/Scripts/osrOCSProvisioning/_syncIMEnabledFromCSO.pl

root     24645 16640  0 14:10 pts/10   00:00:00 grep 23190

 

You will see the full command that is running – in this case a particular perl script. Note that you may also find your grep command in the list … depends a bit on timing if it shows up or not.

You may need to restart a service to clear something that has a memory leak. You may need to stop the process outside of the service control (e.g. stopping the sendmail service doesn’t shut down all current threads). To stop a process, use kill PID# … this is basically asking a process nicely to stop. It will clean up its resources and shut down cleanly. use ps –efww to see if the process is still running. If it still is, use kill -9 PID# which is not asking nicely. Most things to which a process is connected will clean up their own resources after some period of client inactivity (i.e. you aren’t causing a huge number of problems for someone else by doing this) but it is cleaner to use kill without the “do it NOW!!!” option first.

Tail and Grep – Tail is a command that outputs the last n lines of a file. It has a parameter that outputs new lines as they get appended to the file. On *n?x systems, you can use tail –F /path/to/file.xtn and lines will be output as they show up. This is particularly useful on log files where the system is continually adding new info at the bottom of the file. We put Windows ports of these utilities on our Windows servers – but the Windows port of tail does not support –F (there’s a good reason that has to do with the difference between Unix-like and Windows file systems). You can use tail –f instead – if the log file rolls (gets moved to another file and a new file is started) you won’t continue to get output like you will with –F … but you can ctrl-c to terminate the tail & start it again to start seeing the new lines again.

Grep is a command line search program. You can use grep to find lines in a file containing a string (or regex pattern, but learning regex is a question for LMGTFY.com) – to find all of the mail addressed to or from me in a sendmail log, grep –i rushworth /var/log/maillog – the dash i means case insensitive search.

Grep will also search piped input instead of a file – this means you can send the output of tail to grep and display only the lines matching the pattern for which you search.

tail -f /var/log/maillog | grep –i rushworth will output new lines of the maillog as they come in, but only display the ones with my name.

VIM – The non-visual text editor is vim – which is usually invoked using ‘vi’, but vi is an actual program that is like but not exactly the same as vim (vim is improved vi). The vim installation contains a very nice tutorial – invoked by running vimtutor

VIM has both a command mode and an editing mode. When in command mode, different keys on the keyboard have different functions. There are “quick reference” guides and “cheat sheets” online for vim – most people I know have a quick ref guide or cheat sheet taped next to their computer for quite some time before vim commands become well known.

History – Linux maintains a history of commands run in a session. This spans logons (you’ll see commands run last week even through you’ve logged on and off six times between then) but when there are multiple sessions for the same user, there can be multiple history files. Which is all a way of saying you may not see something you expect to see, or you may see things you don’t expect. The output of history shows the command history for the current logon session. You can pipe the output to grep and find commands in the history – for example, if you don’t remember how to start a service, you can use history | grep start and get all commands that contain the string start

[lisa@server855 ~]# history | grep start

7  service ibmslapd start

15  service ibmslapd restart

42  service ibmslapd start

56  service ibmslapd restart

71  service ibmslapd start

95  service ibmslapd start

107  service ibmslapd start

115  service ibmslapd restart

289  service ibmslapd start

303  service ibmslapd start

408  service ibmslapd start

419  service ibmslapd start

430  service ibmslapd start

443  service ibmslapd start

If a command fails, it will still be in the history (all of my typo’s are in there!), but if you see the same command a number of times … it’s probably correct. You can copy/paste the command if you need to edit it to run (or even to run it as-is). You can run the exact command again by typing bang followed by the line number of the history output (!115 with the history above would re-run “service ibmslapd restart”).

Symbolic Links

Linux symbolic links are nothing like Windows shortcuts, although I see people saying that. Shortcuts are independent files that contain a path to the referenced file. Linux sym links are just pointers to the inode for the file. They are the file, just allowing it to be used in a different location. This is a bit like memory addressing in programming — anything that reads from the memory address will get the same data, and anything that writes to the memory address. When you do a long list (ls -al or just ll), you will see both the file name and the file to which it points:

lrwxrwxrwx 1 root root 19 Aug 17 13:54 ljrtest -> /tmp/dnsexit-ip.txt

The “l” at the start of the line indicates that it is a link.