Category: System Administration

Viewing and recording packets using tshark

This time, I’m writing this down so I don’t have to keep looking it up. To display some packet info to the screen while writing a network capture to a file, include the -P option (older versions of tshark used -S)

2021-04-18 13:58:58 [lisa@server ~]# tshark -f "udp port 123" -w /tmp/ntpd.cap -P
Running as user "root" and group "root". This could be dangerous.
Capturing on 'enp0s25'
1 0.000000000 10.x.x.x → x.x.x.18 NTP 90 NTP Version 4, client
2 3.898916081 10.x.x.x → x.x.x.199 NTP 90 NTP Version 4, client
3 7.898948128 10.x.x.x → x.x.x.20 NTP 90 NTP Version 4, client
4 7.928749596 x.x.x.20 → 10.x.x.x NTP 90 NTP Version 4, server
5 9.898958577 10.x.x.x → x.x.x.76 NTP 90 NTP Version 4, client
6 9.949450324 x.x.x.76 → 10.x.x.x NTP 90 NTP Version 4, server
7 10.898981132 10.x.x.x → x.x.x.185 NTP 90 NTP Version 4, client
8 11.009163093 x.x.x.185 → 10.x.x.x NTP 90 NTP Version 4, server

Console access from virsh

I had a whole host of problems that were eventually resolved by rebooting the physical server … but, in the process of trying to figure out exactly what was wrong, I wanted to console into the virtual machines from the physical server. Using “virsh console vmname” should have worked … but it didn’t. Turns out you’ve got to enable a service on each guest before you’re able to console in from the physical server. To do so, run:

systemctl enable serial-getty@ttyS0.service

And, if you want to connect in *right now*, also start the service:

systemctl start serial-getty@ttyS0.service

Now, running “virsh console vmname” doesn’t appear to do much … but, if you hit the enter key, you’ll get a logon prompt for the VM.

Linux: Identifying Large Packages

The disk filled up on our primary server, and there wasn’t anything obvious like a decade worth of log files to clean up. I had to resort to uninstalling ‘stuff’ (it was, after all, installing ‘stuff’ that created the problem … tons of X11-related stuff for troubleshooting purposes). There is a way to list installed packages by size:

 

rpm -qia|awk '$1=="Name" { n=$3} $1=="Size" {s=$3} $1=="Description" {print s " " n }' |sort -n

MTU Probing

We’ve had a number of very strange network problems lately — Zoneminder cannot talk to cameras, clients veg out talking to Myth, Twonky is non-functional (even the web page — you get enough of the header to have a title, but the page just hangs, Scott cannot get to our Discourse site. And, more frustratingly, he cannot SSH to some of our hosts. Using “ssh -v” and throwing on a bunch of flags to not attempt key auth (-o PasswordAuthentication=yes -o PreferredAuthentications=keyboard-interactive,password -o PubkeyAuthentication=no) and his connection still hung. But, at least, I could see something. The last thing the SSH connection reported is:

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

Which I’ve seen before … fortunately when I had a great Unix support guy working in the same office building that I did. Who let me stop over and bounce really oddball problems off of him. He told me to enable mtu probing.

echo 1 >/proc/sys/net/ipv4/tcp_mtu_probing

And, if that doesn’t work, use “echo 2”. Which …. yeah, wouldn’t have been any of my first thirty guesses. Cloudflare published a good article on what exactly MTU path discovery is, and I can RTFM enough to figure out what I’ve set here. But no idea what’s got a smaller MTU than our computers.

 

tcp_mtu_probing - INTEGER
	Controls TCP Packetization-Layer Path MTU Discovery.  
	  0 - Disabled
	  1 - Disabled by default, enabled when an ICMP black hole detected
	  2 - Always enabled, use initial MSS of tcp_base_mss.

PIP SSL Error

Upgraded pip today, and I pretty quickly regretted it. SSL Error attempting to install anything from the Internet (and, amazingly, some things where I downloaded the wheel file). The answer is to downgrade PIP until you hit a version that doesn’t have the error. Annoying. Not sure what the latest rev I could have used was — going back one level and getting the error in loop was more time than I could devote to the project, so I just jumped back six months. Had success with 20.0.2 and left working alone.

Everything from 20.3.1 through 21.0.1 has this failure:

D:\tmp\5\pip>pip install basic_sftp
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLError(1, ‘[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)’))’: /simple/basic-sftp/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLError(1, ‘[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)’))’: /simple/basic-sftp/
WARNING: You are using pip version 20.3.1; however, version 21.0.1 is available.
You should consider upgrading via the ‘c:\programs\anaconda3\python.exe -m pip install –upgrade pip’ command.

SCP From Solaris to RHEL?

Evidently you cannot just scp files from an old Solaris box when you’re on a RHEL/CentOS system … there’s an incompatibility between them that requires you to (1) install scp1 on the Solaris server {not likely in a prod environment} or (2) use sftp to transfer the files.

 

Server1: Red Hat Enterprise Linux Server release 7.6 (Maipo)
Server2: Solaris 5.9

lisa@server1:~/$ scp lisa@server2:/data/stuff/file1.txt ./input/
lisa@server2’s password:
scp: warning: Executing scp1.
scp: FATAL: Executing ssh1 in compatibility mode failed (Check that scp1 is in your PATH).

Fedora — Disabling IPv6

Since it’s the third time I’ve had to do this so far this year, I’m going to write down how I disable IPv6 in Fedora. Add these lines to /etc/sysctl.conf

[lisa@server~]# grep ipv6 /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1

Then load the sysctl settings (sysctl -p) or reboot.

Without IPv6, if you do X-redirection, you may get an error indicating the redirection was refused. In journalctl, there’s an error “error: Failed to allocate internet-domain X11 display socket”. Evidently you’ve got to configure sshd to use IPv4 by setting “AddressFamily inet” in /etc/ssh/sshd_config

[lisa@server~/]# grep AddressFamily /etc/ssh/sshd_config
AddressFamily inet

 

Fedora – Why were my packets dropped?

We’ve been seeing dropped packets on one of our servers — that usually means more data is coming in than can be processed, but it’s nice to confirm rather than guess. The command “netstat -s” displays summary statistics that are nicely grouped into causes:

TcpExt:
16 invalid SYN cookies received
88 resets received for embryonic SYN_RECV sockets
18 packets pruned from receive queue because of socket buffer overrun
2321 ICMP packets dropped because they were out-of-window
838512 TCP sockets finished time wait in fast timer