Filebeat – No Harvesters Starting

Using filebeat-7.17.4, we have seen instances where no harvesters will start and no IP communication is established with the logstash servers. Stopping the filebeat service, confirming the process and any associated network ports are closed, and then starting the service does not restore communication. In this situation, we have had to restart the ​logstash​ servers and immediately begin to see harvesters spin up in the log files:

2022-09-15T12:02:20.018-0400    INFO    [input.harvester]       log/harvester.go:309    Harvester started for paths: 
[/var/log/network/network.log /opt/splunk/var/log/syslog-ng/*/*.log]       
{"input_id": "bf04e307-7fb3-5555-87d5-55555d3fa8d6", "source": "/var/log/syslog-ng/",
 "state_id": "native::2228458-65570", "finished": false, "os_id": "2225548-64550", "old_source": 
"/var/log/syslog-ng/", "old_finished": true, "old_os_id": "2225548-64550", 
"harvester_id": "36555c83-455c-4551-9f55-dd5555552771"}

Logstash – Setting Config with Environment Variables

I took over management of an ElasticSearch environment that has a lot of configuration inconsistencies. Unfortunately, the previous owners weren’t the ones who built the environment either … so no one knew why ServerX did one thing and ServerY did another. Didn’t mess with it (if it’s working, don’t break it!) until we encountered some users who couldn’t find their data — because, depending on which logstash server information transits, stuff ends up in different indices. So now we’re consolidating configurations and I am going to pull the “right” config files into a git repo so we can easily maintain consistency.

Except … any repository becomes in scope for security scanning. And, really, typing your password in clear text isn’t a wonderful plan. So my first step is using environment variables as configuration parameters in logstash.

The first thing to do is set the environment variables somewhere logstash can use them. In my case, I’m using a unit file that sources its environment from /etc/default/logstash

Once the environment variables are there, enclose the variable name in ${} and use it in the config:

Logstash Config

Restart ElasticSearch and verify the pipeline(s) have started successfully.

Finding PCI Devices

You can use dmidecode to list all sorts of information about the system — there is a list of device types that you can use with the “-t” option

   Type   Information
      0   BIOS
      1   System
      2   Baseboard
      3   Chassis
      4   Processor
      5   Memory Controller
      6   Memory Module
      7   Cache
      8   Port Connector
      9   System Slots
     10   On Board Devices
     11   OEM Strings
     12   System Configuration Options
     13   BIOS Language
     14   Group Associations
     15   System Event Log
     16   Physical Memory Array
     17   Memory Device
     18   32-bit Memory Error
     19   Memory Array Mapped Address
     20   Memory Device Mapped Address
     21   Built-in Pointing Device
     22   Portable Battery
     23   System Reset
     24   Hardware Security
     25   System Power Controls
     26   Voltage Probe
     27   Cooling Device
     28   Temperature Probe
     29   Electrical Current Probe
     30   Out-of-band Remote Access
     31   Boot Integrity Services
     32   System Boot
     33   64-bit Memory Error
     34   Management Device
     35   Management Device Component
     36   Management Device Threshold Data
     37   Memory Channel
     38   IPMI Device
     39   Power Supply
     40   Additional Information
     41   Onboard Devices Extended Information
     42   Management Controller Host Interface


[lisa@fedora ~/]# dmidecode -t 9

Handle 0x0024, DMI type 9, 17 bytes
System Slot Information
Designation: Slot6
Type: 32-bit PCI
Current Usage: In Use
Length: Short
ID: 6
3.3 V is provided
Opening is shared
PME signal is supported
Bus Address: 0000:0a:02.0

The “Bus Address” value corresponds to information from lspci:

[lisa@fedora ~/]# lspci | grep “0a:02.0”
0a:02.0 Multimedia video controller: Conexant Systems, Inc. CX23418 Single-Chip MPEG-2 Encoder with Integrated Analog Video/Broadcast Audio Decoder

Recipe – Speculoos

  • 4 cups flour , sifted
  • 1½ cup brown brown sugar
  • 1 cup butter (at room temperature)
  • 3 eggs
  • 1 teaspoon baking powder
  • 1 tablespoon ground cinnamon
  • ½ teaspoon ground ginger
  • ½ teaspoon ground nutmeg
  • ½ teaspoon ground cloves
  • ¼ teaspoon ground cardamom
  • ¼ teaspoon ground white pepper
  • ¼ teaspoon ground anise
  • ¼ teaspoon salt

Mix the flour and baking powder together.

Mix the butter with sugar, salt and spices. Add the eggs one by one and mix well.

Gradually add the flour mixture and stir.

Cover the dough with plastic wrap and refrigerate for 12 hours.

Preheat oven to 375 F / 190 C.

Cut the dough into 4 equal pieces.

Thoroughly dust the work surface with flour and the rolling pin. Roll the first piece dough to a thickness of ¼ inch

Cut the dough with a knife or a cookie cutter and use the wooden or silicone mold to make some prints on the speculoos.

Place the speculoos on a baking sheet lined with parchment paper and bake for about 10 minutes.

Allow to cool for a few minutes, then place on a cookie rack to cool.

Bookshelf Adventure

We’ve been looking for bookshelves for a long time — both Scott and I have a lot of books, and Anya has an ever growing collection of books. We found about a dozen shelves — cantilever metal library bookshelves — and paid six dollars for them all. Basically, the shelves cost our labor and fuel to remove them from the site.

Now, that was a lot of work. We spent two days loading the truck with shelf bits — they used two 15′ box trucks to move the shelves in, but we managed to pack it all quite densely and got all of the shelves packed into the pickup truck bed in two trips. When we counted them all, there are 16 double-sided shelves and a single sided shelf.

ElasticSearch – Listing Snapshots in AWS S3

To view the snapshots held in AWS, you should be able to use Kibana. From “Management” navigate to “Snapshot and Restore” and look at the list of snapshots. We, however, get a timeout attempting to view the snapshots. Instead, use the _snapshot ES API endpoint to get the name of the repository:

Then use the name to create the ES API URI to get a list of snapshots in the repository – GET _snapshot/*?verbose=false – you will get a list of snapshots, which indices are included in each snapshot, and a state (SUCCESS or FAILED).

Heritage Turkeys

In addition to growing open pollinated, heirloom vegetables — we’ve got a flock of heritage turkeys. These guys are Black Spanish turkeys. Unlike the broad-breasted turkeys raised commercially today, they walk around and do turkey things all day. They are all waiting by the gate as we walk over to the poultry pasture, and there are always a few turkeys following us around if we’re working in their area.

The two males we have from last year were amazing with the little poults this Spring. They’d take a share of poults and snuggle them at night to keep them warm. They’d march around them as the little ones pecked around during the day. Even now that the younger turkeys are almost fully grown, the older turkeys stand guard and make sure everyone gets access to food and water. Watching the adult turkeys with the younger ones has been right educational, and I am eager to hatch some of our own poults next year!