Category: Technology

WebLogic LDAP Authentication

Configuring an LDAP Authentication provider in WebLogic (version 11g used in this documentation)

  • In configuring LDAP authentication, I add a new authentication provider but continue to use the local provider for the system account under which WebLogic is launched. Partially because I don’t really use WebLogic (there’s an Oracle app with its own management site that runs within WebLogic – very small number of users, so our configuration is in no way optimized), but partially because using a network-sourced system account can prevent your WebLogic instance from launching. If your config isn’t right, or if the network is down, or a firewall gets in the way, or the LDAP server is down …. Your WebLogic fails to launch because its system ID is not validated.

WebLogic Configuration

Lock & Edit the site so we can make changes. On the left-hand pane, scroll down & find Security Realms

Go into your realm, select the “providers” tab. Supply a name for the provider (I included “LDAP” in the name to ensure it was clear which provider this was – may even want to specify something like “CompanyXLDAPAuthProvider”)

Select type “LDAPAuthenticator” for generic LDAP (I was using Sun DSEE, and moved to Oracle OUD without changing the authenticator type). Click OK to create.

Change the control flag on your default authenticator. Click the hyperlink for the default provider. On the “Common” tab, change the “Control Flag” to “SUFFICIENT” and save.

Click the hyperlink for the newly created provider. On the “Common” tab, change the “Control Flag” to “SUFFICIENT” and save.

Select the “Provider specific” tab.

Connection

Host:     <your LDAP server>

Port:      636

Principal:             <Your system account, provided when you request access to the LDAP directory>

Credentials:        <Your system account password>

Confirm Credentials:       <same as credentials>

SSLEnabled:        Check this box (for testing purposes, i.e. if you are unable to connect with these instructions as provided, you can set the port to 389 and not check this box to help with troubleshooting the problem. But production authentication needs to be done over SSL)

Users

User Base DN:    <get this from your LDAP admin. Ours is “ou=people,o=CompanyX”)

All User Filter:    (&(objectClass=inetOrgPerson))

For applications with a single group restricting valid users, you can use the filter: (&(objectClass=inetOrgPerson)(isMemberOf=cn=GroupNameHere,ou=groups,o=CompanyX))

Users from name filter:  (&(uid=%u)(objectClass=inetOrgPerson))

User Search Type:                           subtree (onelevel may be fine, but verify with your LDAP administrator)

User Name Attribute:                     uid

User Object Class:                           inetOrgPerson

Use Retrieved User Name as Principal – I didn’t select this, don’t really know what it does

Groups

Group Base DN:               <another one to get from your LDAP admin. Ours is “ou=groups,o=CompanyX”>

All Groups Filter:              (&(objectClass=groupOfUniqueNames))

If your group names all have the same prefix, you could limit “all” groups to just your groups with a filter like (&(objectClass=groupOfUniqueNames)(cn=MyApp*))

Group from name filter: (&(cn=%g)(objectclass=groupofuniquenames))

Group search scope:                      subtree (again, onelevel may be fine)

Group membership searching:    <We select ‘limited’ because there are no nested groups in the LDAP directories. If you need to resolve nested group memberships, this and the next value will be different>

Max group membership search level:      0

Ignore duplicate membership:     Doesn’t really matter as we don’t have duplicates. I left this unchecked.

Static groups

Static group Attribute name:       cn

Static group Object Class:             groupOfUniqueNames

Static Member DN Attribute:       uniqueMember

Static Group DNs from Member filter:     (&(uniquemember=%M)(objectclass=groupofuniquenames))

Dynamic Groups              this section is left blank/defaults as we don’t use dynamic groups

General

Connection Pool Size:     Ideal value dependent on your anticipated application load – default of 6 is a good place to start.

Connect timeout:             Default is 0. I don’t know if this is something particular to WebLogic, but I generally use a 15 or 30 second timeout. If the server hasn’t responded in that period, it is not going to respond and there’s no need to hang the thread waiting.

Connection Retry Limit: Default is 1, this should be sufficient but if you see a lot of connection errors, either increase the connect timeout or increase this retry limit

Parallel Connect Delay:  0 (default) is fine

Result time limit:              0 (default) is OK. On my the LDAP server, there is no time limit for searches. Since WebLogic is making very simple searches, you could put a limit in here to retry any search that takes abnormally long

Keep Alive Enabled:         Please do not enable keep alive unless you have a specific need for it. Bringing up a new session uses slightly more time/resources on your app server than re-using an existing connection but that keep alive is a LOT of extra “hey, I’m still here” pings against the LDAP servers

Follow Referrals:              Un-check this box unless your LDAP admin tells you referrals are in use and should be followed.

Bind Anonymously on referrals:  Leave unchecked if you are not following referrals. If referrals are used and followed – ask the LDAP admin how to bind

Propagate cause for logon exception:      I check this box because I *want* the ugly LDAP error code that explains why the logon failed (49 == bad user/password pair; 19 == account locked out). But no *need* to check the box

Cache Related Settings:  This is something that would require more knowledge of WebLogic than I have ?

If you enable caching, you may not see changes for whatever delta-time is the cache duration. So, the defaults of enabling cache & retaining it for 60 seconds wouldn’t really create a problem. If you set the cache duration to one day (a silly setting to make the problem cache can create clear) …. If I logged into your application at 2PM, did a whole bunch of work, went home, came back the next morning & saw my “your password is about to expire” warning … so go out to the password portal and change my password. Reboot, get logged back into my computer …. and try to access your application, I will get told my password is invalid. I could try again, even type what I *know* is my password into notepad & paste it into your app … still not able to log on. My old password, were I to try it, would work … but otherwise I’d have to wait until after 2PM before my new password would work.

Group membership changes could be a problem too – with the same 24 hour cache, if I am a valid user of your application who signs in at 2PM today, but my job function changes tomorrow morning & my access is revoked … I will still have application access until the cache expires. I am not sure if WebLogic does negative caching – basically if I am *not* a user, try to sign in and cannot because I lack the group membership & get an access request approved *really quickly* to become a group member, I may still be unable to access the application until the “Lisa is not a member of group XYZ” cache expires. If WebLogic does not do negative caching, then this scenario is not an issue.

So you might be able to lower utilization on your app server & my LDAP server by enabling cache (if your app, for instance, re-auths the object **each time the user changes pages** or something, then caching would be good). If you are just checking authentication and authorization on logon … probably not going to do much to lower utilization. But certainly keep the cache TTL low (like minutes, not days).

GUID Attribute:  nsUniqueID

Establishing The SSL Trust

For encryption to be negotiated with the LDAP servers, you need to have a keystore that includes the public keys from the CA used to sign the LDAP server cert. Obtain the base 64 encoded public keys either from the PKI admin or the LDAP admin. Place these file(s) on your server – I use the /tmp/ directory since they are no longer needed after import.

From the domain structure section, select: Environment=>Servers and select your server. On the “Configuration” tab, click the keystores sub-tab. If you are not already using a custom trust, you need to change they keystore type to use a custom trust (and specify a filename in a path to which the WebLogic account has access – keystore type is JKS and the password is whatever you are going to make the keystore password). If you *are* already using a custom trust, just record the file name of the custom trust keystore.

Use keytool to import the CA keys to the file specified in the custom trust. The following examples use a root and signing CA from my company, the CA chain which signs our LDAP SSL certs.

./keytool -import -v -trustcacerts -alias WIN-ROOT -file /tmp/WIN-ROOT-CA.b64 -keystore /path/to/the/TrustFile.jks -keypass YourKeystorePassword -storepass YourKeystorePassword

./keytool -import -v -trustcacerts -alias WIN-WEB -file /tmp/WIN-WEB-CA.b64 -keystore /path/to/the/TrustFile.jks -keypass YourKeystorePassword -storepass YourKeystorePassword

*** Under advanced, I had to check off “Use JSSE SSL” for SSL to work. Without that checked off, I got the following error in the log:

####<Feb 23, 2018 10:11:36 AM EST> <Notice> <Security> <server115.CompanyX.com> <AdminServer> <[ACTIVE] ExecuteThread: ’12’ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<WLS Kernel>> <> <58b1979606d98df5:292a2ff6:161c336d0ba:-8000-0000000000000007> <1519398696289> <BEA-090898> <Ignoring the trusted CA certificate “CN=WIN-WEB-CA,DC=CompanyX,DC=com”. The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>

####<Feb 23, 2018 10:11:36 AM EST> <Notice> <Security> <server115.CompanyX.com> <AdminServer> <[ACTIVE] ExecuteThread: ’12’ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<WLS Kernel>> <> <58b1979606d98df5:292a2ff6:161c336d0ba:-8000-0000000000000007> <1519398696289> <BEA-090898> <Ignoring the trusted CA certificate “CN=WIN-Root-CA”. The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>

An alternate solution would be to update your WebLogic instance – there are supposedly patches, but not sure which rev and it wasn’t worth trial-and-erroring WebLogic patches for my one WebLogic instance with a dozen users.

Whew, now save those changes. Activate changes & you will probably need to restart your WebLogic service to have the changes go into effect. You can go into the roles & add LDAP groups as — specifically, I added our LDAP group’s CN to the administrators WebLogic role.

Microsoft Teams – Creating A New Team

Anyone can create a Team space – this makes Teams an amazing resource for collaboration because you have all of the features of Teams without filling out a request form, writing a business justification, and waiting for someone to complete your request. Whether you want to call it a quote from the Spiderman comics, Churchill, or the Decrees of the French National Convention … responsibility follows inseparably from great power.

The first consideration is should you create a Team? Teams is an amazing platform for interactive communication, but not all communication is meant to be interactive and collaborative. If you want to broadcast information to thousands of people (and maybe get a little feedback too), then a Stream site may be a better choice. If you want to solicit feedback about a specific topic and analyze the results, a Forms questionnaire or SharePoint form will likely better suit your needs. If you want to share documents, OneDrive for Business or a SharePoint site may be more appropriate. But if much of your content warrants responses, you want to increase collaboration, you share documents and Planner boards and OneNote notebooks … then you probably want a Teams space.

Can my Team have too many members? Well, from a technical perspective … no. There’s a limit to the number of members you can add to a team – the service won’t let you add too many people. Practically, though, the question isn’t if there are too many members but rather if the information stored in the Teams space is relevant to the individuals. Maybe you’ve got a topic that fifteen hundred people should be discussing – the information helps them do their job, their input helps others. In that case, a team of fifteen hundred people isn’t too many. But if I add thirty people to my Team space and the information is only relevant to eight of them … then I’ve got too many members of my team.

Once you’ve decided that a Teams is a great place to host your collaborative efforts and identified the people who will find the information relevant, here are some “best practice” guidelines for creating and managing your Team.

Click on “Join or create a team” at the bottom of your Teams list.

The Teams carousel will be displayed – search your organization’s public teams to make sure there’s not already one out there doing exactly what you want. At the time of writing, this is a starts with search so searching for “Falcon” will not find “Project Golden Falcon”. To create a new team, click “Create team”.

When creating a Team, the first step is to create a name. Team names do not have to be unique, but it will be confusing for members if they have six “Engineering” teams in their list. Use something descriptive. Filling in the Team description will help members identify the purpose of the Team space too. Click “Next” and optionally add team members.

After your team is created, add another owner. While members can perform most functions within a Team space, there are a few rights limited to Team owners. Adding another owner now ensures you’ve got back-up when you go on holiday or are otherwise unavailable.

Click the hamburger menu next to your team name and select “Manage Team”.

You can add additional members here. And click the drop-down next to any member you wish to become an owner and select “Member” – voila, another owner.

On the “General” channel, add tools and resources that are frequently used – that might be a link to a vendor’s web site (in the Team where we discuss updates and issues with a vendor’s product, having a link to the vendor’s support site is really helpful) or a Planner board to keep track of tasks <ref out to ‘did you know’ on adding the auto-created ones!>. Click the “General” channel then click the + next to the channel’s tabs.

You’ll be presented with a list of resources you can add to your Teams space.

To separate discussions into different channels, click the hamburger menu next to your Team name and select “Add channel”. We will create a new channel for different projects and sub-groups to avoid confusion and information overload.

Active Directory Federation Services (ADFS) Relying Party Trust Cert Expiry

At work, we received a critical ticket for an application that was unable to authenticate to ADFS. Nothing globally wrong – other applications are authenticating. A long call later, we discovered that the app’s certificate has expired. Why would the application not monitor their certificate expiry dates?? That’s an excellent question, but not one over which I have any control.

I can monitor their certs on our side. So I wrote a quick powershell script to grab certificates from the relying party trusts and alerts us if any certs will be expiring in the next 30 days. It has to run on the ADFS server – I’d love to get it moved to the automation server in the future. I expect get-adfsrelyingpartytrust returns disabled agreements. I want to filter out disabled agreements.

Git Pull Requests

I have finally run through the process of submitting a pull request to suggest changes to a Git repository. Do the normal ‘stuff’ either to make a new project or to clone an existing project to your computer. Create a new branch and check out that branch.

C:\ljr\git>git clone https://github.com/ljr55555/SampleProject

Cloning into ‘SampleProject’…

remote: Counting objects: 4, done.

remote: Compressing objects: 100% (3/3), done.

remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0

Unpacking objects: 100% (4/4), done.

C:\ljr\git>cd SampleProject

C:\ljr\git\SampleProject>git branch newEdits

C:\ljr\git\SampleProject>git checkout newEdits

Switched to branch ‘newEdits’

Make some changes and commit them to your branch

C:\ljr\git\SampleProject>git add helloworld.pl

C:\ljr\git\SampleProject>git commit -m “Added hello world script”

C:\ljr\git\SampleProject>git push origin newEdits

Counting objects: 3, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (3/3), done.

Writing objects: 100% (3/3), 408 bytes | 408.00 KiB/s, done.

Total 3 (delta 0), reused 0 (delta 0)

To https://github.com/ljr55555/SampleProject

 * [new branch]      newEdits -> newEdits

On the GitHub site, click the “new pull request” button. Since you select the two branches within the pull request, it doesn’t seem to matter which branch’s “Pull request” tab you select.

Select the source branch and the one with your changes. Verify you can merge the branches (otherwise you’ve got a problem and need to resolve conflicts). Review the changes, then click “Create pull request”

Here’s another place for comments – comments on the pull request, not the commit comments. Click “Create pull request”.

Click “Create pull request” and you’ve got one! Now what do we do with it (i.e. if you’re the repository owner and receive a pull request). If you check the “Pull request” tab on your project, you should see one now.

Click on it to explore the changes that have been made – the “Commits” tab will have the commits, and the “Files changed” tab will show you the specific changes that have been made.

You could just comment and close the pull request (if, for instance, there was a reason you had not implemented the project that way and do not wish to incorporate the changes into your master branch). Assuming you do wish to incorporate the code, there are a couple of ways you can merge the new code into your base branch. The default is generally a good, or read the doc at https://help.github.com/articles/about-pull-request-merges/

Select the appropriate merge type and click the big green button. You have an opportunity to edit the commit message at this point, or just click “Confirm merge”

Voila, it is merged in. You can write some comment to close out the pull request.

There is a notification that the request was completed and the branch can be deleted.

And the project no longer has any open pull requests (you can remove the “is open” filter and see the request again).

And finally, someone should delete the branch. Is that the person who created the branch? Is that the person who maintains the repository? No idea! I’d delete my own, to keep things tidy … but I wouldn’t be offended if the maintainer deleted it either.

 

Drone Army

Over the weekend, when it was negative five degrees, our neighbor’s power went out in the middle of the night. Some trees along the line grew into the power lines and had been abrading the line for some time, and a handful of arborists had to come out and try to trim the tree back. In the dark. At negative five degrees. Not the most fun job I could imagine, and the ironic this is it was the same team that had been out in the summer to clear trees along a stretch of the power lines a bit farther down.

The problem, it seems, is that it’s terribly time consuming to have arborists walking along the line to see where things actually need to be cut. Instead they just hit every section once per unit time. Sometimes that’s a quick couple branches snipped in a hardwood grove. Sometimes that’s serious maintenance in softwood groves. And sometimes delta-time is too long for, say, our line of pine trees. And sometimes the team doesn’t do a particularly good job of trimming the trees.

Made me wonder about having drones fly along the line – you’d still need someone to drive out, and I’d recharge the batteries in the van/truck so they’d be ready to go when I got to the next site. A single person flying a drone over a stretch of power lines could generate more realistic work orders for the arborists – skip the bits that didn’t grow much, realize these pine trees are endangering the lines before you had to call out a crew on Sunday night. They could also run through the same line post-maintenance and verify the work was done well.

Home Security Drone

We’ve conceptualized home security drones for some time with autonomous programming that instructs the drones to return to a charging station when their batteries become depleted. Feed the video back to a platform that knows what the area should look like and alert on abnormalities.

The idea of a drone patrol is interesting to me because optimizing the ‘random walk’ algorithm to best suit the implementation is challenging. The algorithm would need to be modified to account for areas that other drones recently visited and allow weighting for ease of ingress (i.e. it’s not likely someone will scale a cliff wall to infiltrate your property. A lot of ‘intrusions’ will come through the driveway). Bonus points for a speaker system that would have the drone direct visitors to the appropriate entrance (please follow me to the front door) — a personal desire because delivery people seem to believe both our garage and our kitchen patio are the front door.

This is a great security solution when it’s unique, but were the idea to be widely adopted … it would suck as a home security implementation. Why? Drones with video feeds sound like a great way to deter trespassing. But drones have practical limitations. Home break-ins would be performed during storms. Or heavy snowfall. Or …

What if the drone charging base has wheels – during adverse weather, the drone can convert itself into an autonomous land vehicle. I’d probably include an additional battery in the base as the wheeled vehicle traversing land would use more energy. And there would be places a wheeled vehicle could not travel. The converted drone would be able to cover some of the property, and generally the area closest to the structures could be traversed.

Spectre & Meltdown

The academic whitepapers for both of these vulnerabilities can be found at https://spectreattack.com/ — or El Reg’s article and their other article provide a good summary for those not included to slog through technical nuances. There’s a lot of talk about chip manufacturer’s stock drops and vendor patches … but I don’t see anyone asking how bad this is on hosted platforms. Can I sign up for a free Azure trial and start accessing data on your instance? Even if they isolate free trial accounts (and accounts given to students through University relationships), is a potential trove of data worth a few hundred bucks to a hacker? Companies run web storefronts that process credit card info, so there’s potentially profit to be made. Hell, is the data worth a few million to some state-sponsored entity or someone getting into industrial espionage? I’m really curious if MS uses the same Azure farms for their hosted Exchange and SharePoint services.

While Meltdown has patches (not such a big deal if you’re use cases are GPU intensive games, but does a company want a 30% performance hit on business process servers, automated build and testing machines, data mining servers?), Spectre patches turn IT security into TSA regulations. We can make a patch to mitigate the last exploit that occurred. Great for everyone else, but doesn’t help anyone who experienced that last exploit. Or the people about to get hit with the next exploit.

I wonder if Azure and AWS are going to give customers a 5-30% discount after they apply the performance reducing patch? If I agreed to pay x$ for y processing capacity, now they’re supplying 0.87y … why wouldn’t I pay 0.87x$?

3D Print Server – OctoPrint

When we started setting up our 3D printer, I installed Cura on my laptop … but I don’t want to leave my laptop in the office & hooked up to the printer for a day or two. We could install Cura on the server and use it to print, but we’d also need to use something like xvnc so we could remotely initiate a print job and not need to stay connected to a redirected X session for a day or two. Thus began the quest for a server-based 3D printer controller. I think we’re going to use OctoPrint on our Fedora server.

There are a few prerequisities: python, python-pip, and python2-virtualenv, and git-core (well, you can just download/extract the project … but having a git client is quicker/easier).

In the directory where you want the OctoPrint folder, run “git clone https://github.com/foosel/OctoPrint.git”

Create a user for octoprint and add that user to the tty and dialout groups.

Create a python virtual environment: virtualenv venv

Install OctoPrint into the new environment: ./venv/bin/python setup.py install

Log into the octoprint service account (interactive logon or su), start a screen session for the server, then start the server with in the screen:

su – myserviceaccount
screen -d -m -S OctoPrintServer
screen -x OctoPrintServer
/path/to/OctoPrint/venv/bin/octoprint

Then access the web service and continue setup – the default port is 5000. My next step is to write an init script so the server will auto-launch on restart … but this is functional enough to start printing.

 

Customer Service And IT Automation

A 3D printer filament manufacturer, MakerGeeks, has been running a series of awesome deals since Black Friday. We placed an order for several of their their “grab bag” packages – which I assume to be production overruns and whatever isn’t selling. We want to make a few large prototypes – if it’s an amalgamation of oddball colours … whatever, it’ll still be functional. We can pay extra to select the colour once we’ve got a finished model file.

A few hours after placing my order, I got a mass e-mail saying essentially “we sold a lot more stuff than we expected, it’s gonna take a while to ship”. Wasn’t buying Christmas presents, so waiting a while … whatever. Two weeks later, I haven’t heard a thing from them. Odd. I sent a quick e-mail asking for someone to verify that my order didn’t get lost or something. And never heard back from them. Waited another week and sent a follow-up.

Checked them out on the BBB site and found out they’ve got a really bad reputation for non-existent customer service And not shipping ‘stuff’. Sent an e-mail to all of the contacts listed on the BBB site (the phone number is unanswered and rolls to a generic message). Another week with no response, and I filed a BBB complaint mostly to increase the number of people saying “these people don’t bother answering e-mail and suck at order fulfillment”.

Additional irony – I’d subscribed to their newsletter when we placed our order. The five weeks of no communication from the company did include an almost daily e-mail with information on their holiday promotion. So they’re not bothering to ship my stuff, but they’re actively soliciting new orders!?!

What bothers me, though, is that a simple automated job would be the difference between initiating a charge-back and waiting for my order to ship. There’s an order database somewhere. Pull a list of all open orders & send a message that says increasingly comforting versions of “we haven’t forgotten about you, we just haven’t gotten to you yet”. If it were me, I’d probably include something like “We currently have outstanding orders for 25,839 KG of filament that we’re working through. The machines are running as fast as they can, and we’re shipping 2,848 KG a day. We want to thank you for your patience as we work through this amazing volume of holiday orders.”. Actual message content is almost irrelevant. The point is a few dozen development hours would be saving orders and improving the company’s reputation.

Instead I get nothing. With no faith that the company will ship me anything ever … and since I don’t want to try disputing a charge six months after it was made (had problems with that before – prepaid a CSA membership through PayPal, waited eight months for the new cycle to start, but I wasn’t on their list and they claimed to have no record of my payment. Tried to dispute it through PayPal and was told the window to dispute the charge was up … but I didn’t know I wasn’t going to be part of the new year until the first delivery!), I presented my communication and their complete lack of response to the credit card company. About 24 hours later, the charge-back was completed.

Ransomware

My company held a ransomware response through experiment recently – and, honestly, every ransomware response I’ve seen has been some iteration of “walk through backups until we find good files”. Maybe use something like the SharePoint versioning to help identify a good target date (although that date may be different for different files … who knows!). But why wouldn’t you attempt a proactive identification of compromised files?

The basis of ransomware is that it encrypts data and you get the password after paying so-and-so a bitcoin or three. Considering that NGO virus authors (e.g. those who aren’t trying to slow down Iran’s centrifuges) are generally interested in creating mayhem. There’s not a lot of disincentive to creating mayhem and making a couple of bucks. I don’t anticipate ransomware to become less prevalent in the future; in fact I anticipate seeing it in vigilante hacking: EntityX gets their files back after they publicly donate 100k to their antithesis organisation.

Since it’s probably not going away, it seems worthwhile to immediately identify the malicious data scrambling. Reverting to yesterday’s backups sucks, but not as much as finding that your daily backups have aged out and you’re stuck with the monthly backup from 01 Nov as your last “good” data set. It would also be good to merge whatever your last good backup is into the non-encrypted files so the only ‘stuff’ that reverts is a worthless scramble of data anyway. Sure someone may have worked on the file this morning and sucks for them to find their work back-rev’d to last night … but again that’s better than everyone having to reproduce their last two and a half months of work.

Promptly identifying the attack: There are routine processes that read changed files. Windows Search indexing, antivirus scanner, SharePoint indexing. Running against the Windows Search index log on every computer in the organisation is logistically challenging. Not impossible, but not ideal either. A central log for enterprise AV software or the SharePoint indexing log, however, can be parsed from the data centre. Scrape the log files for “unable to read this encrypted file” events. Then there are a myriad of actions that can be taken. Alert the file owner and have them confirm the file should be encrypted. Alert the IT staff when more than x encrypted files are identified in a unit time. Check the create time-stamp and alert the file owner for any files that were created prior to encountering them as encrypted.

Restoring only scrambled files: Since you have a list of encrypted files, you have a scope for the restore job. Instead of restoring everything in place (because who has 2x the storage space to restore to an alternate location?!). Restore just the recently identified as encrypted files – to an alternate location or in place. Ideally you’ve gotten user input on the encrypted files and can omit any the user indicated they encrypted too.