The disk filled up on our primary server, and there wasn’t anything obvious like a decade worth of log files to clean up. I had to resort to uninstalling ‘stuff’ (it was, after all, installing ‘stuff’ that created the problem … tons of X11-related stuff for troubleshooting purposes). There is a way to list installed packages by size:
rpm -qia|awk '$1=="Name" { n=$3} $1=="Size" {s=$3} $1=="Description" {print s " " n }' |sort -n
We’ve had a number of very strange network problems lately — Zoneminder cannot talk to cameras, clients veg out talking to Myth, Twonky is non-functional (even the web page — you get enough of the header to have a title, but the page just hangs, Scott cannot get to our Discourse site. And, more frustratingly, he cannot SSH to some of our hosts. Using “ssh -v” and throwing on a bunch of flags to not attempt key auth (-o PasswordAuthentication=yes -o PreferredAuthentications=keyboard-interactive,password -o PubkeyAuthentication=no) and his connection still hung. But, at least, I could see something. The last thing the SSH connection reported is:
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Which I’ve seen before … fortunately when I had a great Unix support guy working in the same office building that I did. Who let me stop over and bounce really oddball problems off of him. He told me to enable mtu probing.
echo 1 >/proc/sys/net/ipv4/tcp_mtu_probing
And, if that doesn’t work, use “echo 2”. Which …. yeah, wouldn’t have been any of my first thirty guesses. Cloudflare published a good article on what exactly MTU path discovery is, and I can RTFM enough to figure out what I’ve set here. But no idea what’s got a smaller MTU than our computers.
tcp_mtu_probing - INTEGER
Controls TCP Packetization-Layer Path MTU Discovery.
0 - Disabled
1 - Disabled by default, enabled when an ICMP black hole detected
2 - Always enabled, use initial MSS of tcp_base_mss.Reminder to self — when you set up a desktop app with OAUTH to use the Google APIs … you have to hit the authorization URL from the computer running the code. That means, for my calendar scraper, that I need to do X-redirection from the server & run the script. Firefox launches & the flow actually completes. Attempting to hit the URL from my computer yields a connection failure to the https://localhost:SomePort at the end of the workflow.
Move token.pickle to backup file, run getCalendarEvents.py with X-redirection so auth can be processed through web form.
I am writing this down because I never manage to remember these two super useful functions that tells you what a variable is.
iLastProcessedTimestamp = 0
with open(‘test.txt’) as f:
iLastProcessedTimestamp = int(f.readline())
print(dir(iLastProcessedTimestamp))
print(type(iLastProcessedTimestamp))
The type function tells you the variable’s class (in this case, int). The dir function tells you the attributes of the variable.
Every once in a while, my Windows server says it has no Internet connection. It does. It can even use the connection. But the little systray icon has a yellow glif and it says there’s no connection. It’s cosmetic, but annoying. I’ve discovered, however, that there’s a “Network Location Awareness” service that can be restarted to re-evaluate the connectivity status.
MS Word has an ‘ignore this error’ thing in the grammar checker that I use fairly regularly — technical writing has syntax that reads as wrong, grammatical errors for impact, or informal writing where I don’t much care for some rules of grammar … I don’t want to turn off the grammar checker, but I do want to stop seeing a squiggly line under a specific sentence that I don’t want to change. Turns out Pylint has something similar:
Upgraded pip today, and I pretty quickly regretted it. SSL Error attempting to install anything from the Internet (and, amazingly, some things where I downloaded the wheel file). The answer is to downgrade PIP until you hit a version that doesn’t have the error. Annoying. Not sure what the latest rev I could have used was — going back one level and getting the error in loop was more time than I could devote to the project, so I just jumped back six months. Had success with 20.0.2 and left working alone.
Everything from 20.3.1 through 21.0.1 has this failure:
D:\tmp\5\pip>pip install basic_sftp
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLError(1, ‘[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)’))’: /simple/basic-sftp/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘SSLError(SSLError(1, ‘[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)’))’: /simple/basic-sftp/
WARNING: You are using pip version 20.3.1; however, version 21.0.1 is available.
You should consider upgrading via the ‘c:\programs\anaconda3\python.exe -m pip install –upgrade pip’ command.
Evidently you cannot just scp files from an old Solaris box when you’re on a RHEL/CentOS system … there’s an incompatibility between them that requires you to (1) install scp1 on the Solaris server {not likely in a prod environment} or (2) use sftp to transfer the files.
Server1: Red Hat Enterprise Linux Server release 7.6 (Maipo)
Server2: Solaris 5.9
lisa@server1:~/$ scp lisa@server2:/data/stuff/file1.txt ./input/
lisa@server2’s password:
scp: warning: Executing scp1.
scp: FATAL: Executing ssh1 in compatibility mode failed (Check that scp1 is in your PATH).
Since it’s the third time I’ve had to do this so far this year, I’m going to write down how I disable IPv6 in Fedora. Add these lines to /etc/sysctl.conf
[lisa@server~]# grep ipv6 /etc/sysctl.conf net.ipv6.conf.all.disable_ipv6=1 net.ipv6.conf.default.disable_ipv6=1 net.ipv6.conf.lo.disable_ipv6=1
Then load the sysctl settings (sysctl -p) or reboot.
Without IPv6, if you do X-redirection, you may get an error indicating the redirection was refused. In journalctl, there’s an error “error: Failed to allocate internet-domain X11 display socket”. Evidently you’ve got to configure sshd to use IPv4 by setting “AddressFamily inet” in /etc/ssh/sshd_config
[lisa@server~/]# grep AddressFamily /etc/ssh/sshd_config AddressFamily inet
In the process of troubleshooting UPNP/DLNA on our MythTV server, I learned that you can send logging verbosity settings while the server is running. Using the mythbackend binary with the –setverbose flag, you can specify logging level. For example:
mythbackend --setverbose http:debug,upnp:debug
What items can you set levels on? It’ll conveniently tell you — “all” or “none” override existing settings, everything else will update the current logging levels (i.e. if I’ve already got http and upnp in debug, I can use “–setverbose audio:debug” to add audio to the list of things in debug mode).
[mythuser@server /var/log/mythtv/]# mythbackend -v help Verbose debug levels. Accepts any combination (separated by comma) of: all - ALL available debug output audio - Audio related messages channel - Channel related messages chanscan - Channel Scanning messages commflag - Commercial detection related messages database - Display all SQL commands executed decode - MPEG2Fix Decode messages dsmcc - DSMCC carousel related messages dvbcam - DVB CAM debugging messages eit - EIT related messages file - File and AutoExpire related messages frame - MPEG2Fix frame messages general - General info gpu - GPU OpenGL driver messages gpuaudio - GPU Audio Processing messages gpuvideo - GPU video rendering messages gui - GUI related messages http - HTTP Server messages idle - System idle messages jobqueue - JobQueue related messages libav - Enables libav debugging media - Media Manager debugging messages mheg - MHEG debugging messages most - Most debug (nodatabase,notimestamp,noextra) network - Network protocol related messages none - NO debug output osd - On-Screen Display related messages playback - Playback related messages process - MPEG2Fix processing messages record - Recording related messages refcount - Reference Count messages rplxqueue - MPEG2Fix Replex Queue messages schedule - Scheduling related messages siparser - Siparser related messages socket - socket debugging messages system - External executable related messages timestamp - Conditional data driven messages upnp - UPnP debugging messages vbi - VBI related messages xmltv - xmltv output and related messages
To disable debugging, use “mythbackend –setverbose none”