Category Archives: Coding

Check Windows Servers Activation Status

I needed a quick way to check activation status of Windows Servers in a domain. This is the solution I came up with using PowerShell to run the slmgr.vbs script for output. I’m not great with PowerShell, and I’m sure this can be cleaned up or made more efficient, but this ‘hack’ worked for me.

$computers = get-adcomputer -filter "OperatingSystem -Like '*Windows Server*' -and Enabled -eq 'True'" | select-object name

foreach($computer in $computers) {
	write-host $computer.name
	Invoke-Command { (cscript /Nologo "C:\Windows\System32\slmgr.vbs" /xpr) -join '' } -ComputerName $computer.name
}

The output will be one of the following, with variation to the edition:

Windows(R), ServerStandard edition:    Windows is in Notification mode
Windows Server(R), ServerDatacenter edition:    The machine is permanently activated.

FileSystemWatcher – LastAccess Not Working

To have System.IO.FileSystemWatcher LastAccess work, the system must have access logging enabled with the following command.

fsutil behavior set DisableLastAccess 0

After setting this, reboot, and you can successfully use the FileSystemWatcher to monitor LastAccess of files (sorta).

Example Code

FileSystemWatcher lWatcher = new FileSystemWatcher(@"C:\windows\temp", "*.*");
                lWatcher.NotifyFilter = NotifyFilters.LastAccess;
                lWatcher.EnableRaisingEvents = true;
                lWatcher.Changed += new FileSystemEventHandler(HandlerWatcherChanged);
                lWatcher.IncludeSubdirectories = true;
            }
        }
        static void HandlerWatcherChanged(object sender, FileSystemEventArgs e)
        {
            Console.WriteLine("[" + DateTime.Now.ToString() + "] ACCESS " + e.FullPath.ToString());


        }

Pennsylvania COVID-19 Data Scraping with Python

Over the last few weeks I’ve been using Python to scrape the Pennsylvania Department of Health’s Coronavirus page. Over time the page has evolved and even split into sub-pages which contain table date of cases, deaths and other statistics.

I’ve decided to put my Python script on GitHub for public consumption. Initially when I had created the script, it was used to send me alerts when the reported numbers changed as there was no set time during the day that the website was updated, so I wanted to set up a cron job to check the website and alert me of new updates.

Below you’ll find the main script. Note that the code below may be out of date, so please check my GitHub repository for the latest. The code changes almost daily as it seems Pennsylvania Dept. of Health changes the structure, or adds new data, to the webpage which throws off my code.

Python Script

import pandas as pd
from bs4 import BeautifulSoup
import requests
import os
import re
url = r'https://www.health.pa.gov/topics/disease/coronavirus/Pages/Cases.aspx'
html_content = requests.get(url).text
lastupdatedfile = "lastupdated.txt"
soup = BeautifulSoup(html_content, "lxml")
stats = soup.find("span",attrs={"class": "ms-rteStyle-Quote"})
#TODO: byte / string issue here on updatecheck
updatecheck = stats.text[stats.text.find("at "):][3:]
if os.path.isfile(lastupdatedfile):
	lastupdate = open(lastupdatedfile).read()
	if lastupdate == updatecheck:
		print("Skipping check, no new update.")
		os._exit(0)
	else:
		print("***UPDATE***\nOld: {}".format(lastupdate))
		print("New: {}".format(updatecheck))
tables = pd.read_html(html_content, header=0)
df = tables[3]
totalCounties = 67
print("Pennsylvania Data ({})".format(updatecheck))
deathsTotal = int(df["Deaths"].sum())
casesTotal  = int(df["Number of Cases"].sum())
mortalityPercent = round((deathsTotal / casesTotal) * 100,2)
reportingTotal = int(df["County"].count())
reportingCases = df["Number of Cases"]
reportingCasesPct = round((reportingCases.count() / totalCounties) * 100,2)
reportingDeathsObj = df.apply(lambda x: True if x['Deaths'] > 0 else False, axis=1)
reportingDeaths = len(reportingDeathsObj[reportingDeathsObj == True].index)
reportingDeathsPct = round((reportingDeaths / reportingTotal) * 100,2)
print("Total Cases: {}".format(casesTotal))
print("Total Deaths: {}".format(deathsTotal))
print("Mortality Rate(%): {}".format(mortalityPercent))
print("Counties Reporting Cases: {}".format(reportingCases.count()))
print("Counties Reporting Cases(%): {}".format(reportingCasesPct))
print("Counties Reporting Deaths: {}".format(reportingDeaths))
print("Counties Reporting Deaths(% of counties reporting cases): {}".format(reportingDeathsPct))
f=open(lastupdatedfile,"w")
f.write(updatecheck)
f.close()
# Add some notification stuff here...

Output Example

Pennsylvania Data (12:00 p.m. on 3/28/2020)
Total Cases: 2751
Total Deaths: 34
Mortality Rate(%): 1.24
Counties Reporting Cases: 56
Counties Reporting Cases(%): 83.58
Counties Reporting Deaths: 13
Counties Reporting Deaths(% of counties reporting cases): 23.21

Login to WordPress from Python

I’ve been trying to learn some Python and have been tinkering with the requests module. Here is how I am able to log into a webpage, such as WordPress.

import requests
 url = "https://techish.net/wp-login.php"
 redirect_to  = "https://techish.net/wp-admin/"
 with requests.Session() as session:
     post = session.post(url, data={
         'log': 'admin',
         'pwd': 'password',
         'redirect_to': redirect_to
         }, allow_redirects=True)
     get = session.get(redirect_to, cookies=post.cookies)
     print(get.text)

Forgotten Scripts

Came across a script I wrote some time ago. Sitting there in my user directory was patch.cmd. As I find my scripts, I’m putting them in Git for historical purposes… and for a good laugh.

What in the actual fuck are you, patch.cmd? –Rich

One thing I learned;  document the damn things.  I had no idea what this was, but I knew it was mine because of some of the commentary in the batch script.
Anyway, this script was used to patch atmfd.dll (see: MS15-078 Bulletin).

@echo off
if NOT %~n0==patch IF NOT %~n0==unpatch goto FAILINIT
reg Query HKLMHardwareDescriptionSystemCentralProcessor | find /i x86 > NUL && set ARCH=32BIT || set ARCH=64BIT
for /f tokens=2* %%a in ('REG QUERY HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersion /v CurrentVersion') do set WINV=%%~b
REM Operating system              Version number
REM ----------------------------  --------------
REM Windows 10 Tech.Preview       10.0
REM Windows Server Techn.Preview  10.0
REM Windows 8.1                    6.3
REM Windows Server 2012 R2         6.3
REM Windows 8                      6.2
REM Windows Server 2012            6.2
REM Windows 7                      6.1
REM Windows Server 2008 R2         6.1
REM Windows Server 2008            6.0
REM Windows Vista                  6.0
REM Windows Server 2003 R2         5.2
REM Windows Server 2003            5.2
REM Windows XP 64-Bit Edition      5.2
REM Windows XP                     5.1
REM Windows 2000                   5.0
REM check to see if goto label is there.  if it's not, then we're not on a supported
REM system or Rich is a dork.
findstr /r /i /c:^:%~n0_%WINV%_%ARCH% %0>nul
if errorlevel 1 (
echo.
echo Looks like this system is not supported.  %ARCH% - Windows - %WINV%
goto EOF
)
goto %~n0_%WINV%_%ARCH%
REM catchall - shouldn't get here.
goto EOF
pause
:patch_6.0_32BIT
goto patch_1_32BIT
:patch_6.0_64BIT
goto patch_1_64BIT
:patch_6.1_32BIT
goto patch_1_32BIT
:patch_6.1_64BIT
goto patch_1_64BIT
:patch_6.2_32BIT
goto patch_2_32BIT
:patch_6.2_64BIT
goto patch_2_64BIT
:patch_6.3_32BIT
goto patch_2_32BIT
:patch_6.3_64BIT
goto patch_2_64BIT
:patch_10.0_32BIT
goto patch_2_32BIT
:patch_10.0_64BIT
goto patch_2_64BIT
REM unpatch routines
:unpatch_6.0_32BIT
goto unpatch_1_32BIT
:unpatch_6.0_64BIT
goto unpatch_1_64BIT
:unpatch_6.1_32BIT
goto unpatch_1_32BIT
:unpatch_6.1_64BIT
goto unpatch_1_64BIT
:unpatch_6.2_32BIT
goto unpatch_2_32BIT
:unpatch_6.2_64BIT
goto unpatch_2_64BIT
:unpatch_6.3_32BIT
goto unpatch_2_32BIT
:unpatch_6.3_64BIT
goto unpatch_2_64BIT
:unpatch_10.0_32BIT
goto unpatch_2_32BIT
:unpatch_10.0_64BIT
goto unpatch_2_64BIT
:patch_1_32BIT
cd %windir%system32
takeown.exe /f atmfd.dll
icacls.exe atmfd.dll /save atmfd.dll.acl
icacls.exe atmfd.dll /grant Administrators:(F)
rename atmfd.dll x-atmfd.dll
echo.
echo Done patching Windows %WINV% (%ARCH%)
echo.
goto EOF
:patch_1_64BIT
cd %windir%system32
takeown.exe /f atmfd.dll
icacls.exe atmfd.dll /save atmfd.dll.acl
icacls.exe atmfd.dll /grant Administrators:(F)
rename atmfd.dll x-atmfd.dll
cd %windir%syswow64
takeown.exe /f atmfd.dll
icacls.exe atmfd.dll /save atmfd.dll.acl
icacls.exe atmfd.dll /grant Administrators:(F)
rename atmfd.dll x-atmfd.dll
echo.
echo Done patching Windows %WINV% (%ARCH%)
echo.
goto EOF
:patch_2_32BIT
REM I'm not using the registry modification - I planned to, but backed out.  I'll
REM stick to just renaming the file for now.
goto patch_1_32BIT
:patch_2_64BIT
REM I'm not using the registry modification - I planned to, but backed out.  I'll
REM stick to just renaming the file for now.
goto patch_1_64BIT
REM unpatch routines
:unpatch_1_32BIT
cd %windir%system32
rename x-atmfd.dll atmfd.dll
icacls.exe atmfd.dll /setowner NT SERVICETrustedInstaller
icacls.exe . /restore atmfd.dll.acl
echo.
echo Done unpatching Windows %WINV% (%ARCH%)
echo.
goto EOF
:unpatch_1_64BIT
cd %windir%system32
rename x-atmfd.dll atmfd.dll
icacls.exe atmfd.dll /setowner NT SERVICETrustedInstaller
icacls.exe . /restore atmfd.dll.acl
cd %windir%syswow64
rename x-atmfd.dll atmfd.dll
icacls.exe atmfd.dll /setowner NT SERVICETrustedInstaller
icacls.exe . /restore atmfd.dll.acl
echo.
echo Done unpatching Windows %WINV% (%ARCH%)
echo.
goto EOF
:unpatch_2_32BIT
REM I'm not using the registry modification - I planned to, but backed out.  I'll
REM stick to just renaming the file for now.
goto unpatch_1_32BIT
:unpatch_2_64BIT
REM I'm not using the registry modification - I planned to, but backed out.  I'll
REM stick to just renaming the file for now.
goto unpatch_1_64BIT
:FAILINIT
echo Script needs to be named patch or unpatch.
:EOF

Get File Count Recursively

I’ve been working on a small tool to aid in removing duplicate files and as I’m going back over my roughed in code, I’m trying to optimize it for some performance gains.

This snippet of code works really well for recursively counting files given a specific path.  I originally found it at StackOverflow and slightly modified to suit my needs.

Sub ProcessFile(ByVal path As String)
        fileCounter += 1
    End Sub
    Sub ApplyAllFiles(ByVal folder As String, ByVal extension As String, ByVal fileAction As ProcessFileDelegate)
        For Each file In Directory.GetFiles(folder, extension)
            fileAction.Invoke(file)
        Next
        For Each subDir In Directory.GetDirectories(folder)
            Try
                ApplyAllFiles(subDir, extension, fileAction)
            Catch ex As Exception
            End Try
        Next
    End Sub

It processes about 27k files in 1.5 seconds on my SSD disk.  I have it running against a NAS with considerably larger amount of files, so I’ll see how well it performs.
In my sub, I use the following to kick it off.

        Dim fileCounter as Long = 0L
        Dim path = "z:"
        Dim ext = "*.*"
        ToolStripStatusLabel1.Text = "Calculating files..."
        stpw.Start()
        Dim runProcess As ProcessFileDelegate = AddressOf ProcessFile
        ApplyAllFiles(path, ext, runProcess)
        stpw.Stop()
        Dim rslts3 As String = String.Format("Total files = {0:n0}. Took {1:n0} ms.", fileCounter, stpw.ElapsedMilliseconds)
        ToolStripStatusLabel1.Text = rslts3.ToString

Graphically speaking, this isn’t much to look at – but the important part is in the ToolStripStatus. I have a timer on my form that updates the latest file count every 15 seconds so that a user would know it’s still working. Interestingly enough, if I update the ToolStripStatus with every single file that is found, it exponentially increases the time it takes to go through the files, so I decided to just update every 15 seconds.

Development Log: Duplicate File Finder

I have thousands of files stored on an external USB attached 1TB drive.  My drive is currently 95% full.  I know I have duplicate files throughout the drive because over time I have been lazy and made backups of backups (or copies of copies) of images or other documents.
Time to clean house.
I’ve searched online for a tool to do the following things, relatively easily and in a decent designed user interface:

  • Find duplicates based on hash (SHA-256)
  • List duplicates at end of scan
  • Give me an option to delete duplicates, or move them somewhere
  • Be somewhat fast

Every tool I’ve used fell short somewhere.  So I decided to write my own application to do what I want.
What will my application do?
Hash each file recursively given a starting path and store the following information into an SQLite database for reporting and/or cleanup purposes.

  • SHA-256 Hash
  • File full path
  • File name
  • File extension
  • File mimetype
  • File size
  • File last modified time

With this information, I could run a report such as the following pseudo report:
Show me a list of all duplicate files with an extension of JPG over a file size of 1MB modified in the past 180 days.

That’s just a simple query, something like:

SELECT fileHash, fileName, filePath, fileSize COUNT(fileHash) FROM indexed_files WHERE fileExtension='JPG' and fileSize > 1024 GROUP BY fileHash HAVING COUNT(fileHash)>1

My application can show me a list of these and make some decisions to allow me to move or delete the duplicates after the query runs.

One problem comes to mind in automating removal or moving duplicates… What if there are more than 1 duplicate file; how do I handle this?

So on to the bits and pieces…

The hashing function is pretty straight-forward in VB.NET (did I mention I was writing this in .NET?).

Imports System.IO
Imports System.Security
Imports System.Security.Cryptography
Function hashFile(ByVal fileName As String)
  Dim hash
  hash = SHA256.Create()
  Dim hashValue() As Byte
  Dim fileStream As FileStream = File.OpenRead(fileName)
  fileStream.Position = 0
  hashValue = hash.ComputeHash(fileStream)
  Dim hashHex = PrintByteArray(hashValue)
  fileStream.Close()
  Return hashHex
End Function
Public Function PrintByteArray(ByVal array() As Byte)
  Dim hexValue As String = ""
  Dim i As Integer
  For i = 0 To array.Length - 1
    hexValue += array(i).ToString("X2")
  Next i
  Return hexValue.ToLower
End Function
Dim path As String = "Z:"
' Insert recursion function here and inside, use the following:
Dim fHash = hashFile(path) ' The SHA-256 hash of the file
Dim fPath = Nothing ' The full path to the file
Dim fName = Nothing ' The filename
Dim fExt = Nothing ' The file's extension
Dim fSize = Nothing ' The file's size in bytes
Dim fLastMod = Nothing ' The timestamp the file was last modified
Dim fMimeType = Nothing ' The mimetype of the file

Ok cool, so I have a somewhat workable code idea here. I’m not sure how long this is going to take to process, so I want to sample a few hundred files and maybe even think about some options I can pass to my application such as only hashing specific exensions or specific file names like *IMG_* or even be able to exclude something.
But first… a proof of concept.

Update: 11/28/2016

Spent some time working on the application.  Here’s a GUI rendition;  not much since it is being used as a testing application.

I have also implemented some code for SQLite use to store this to a database.  Here’s a screenshot of the database.

Continuing on with some brainstorming, I’ve been thinking about how to handle the multiple duplicates.
I think what I want to do is

  • Add new table “duplicates”
  • Link “duplicates” to “files” table by “id” based on duplicate hashes
  • Store all duplicates found in this table for later management (deleting, archiving, etc.)

After testing some SQL queries and using some test data, I came up with this query:

SELECT * FROM file a
WHERE ( hash ) IN ( SELECT hash FROM file GROUP BY hash HAVING COUNT(*) > 1 )

This gives me the correct results as illustrated in the screenshot below.

So with being able to pick out the duplicate files and display them via a query, I can then use the lowest “id” as the base or even the last modified date as the original and move the duplicates to a table to be removed or archived.
Running my first test on a local NAS with thousands of file.  It’s been running about 3 hours and the database file is at 1.44MB.

Update 12/1/2016

I’ve worked on the application off and on over the past few days trying to optimize the file recursion method.  I ended up implementing a faster method than I created above, and I wrote about it here.

Here’s a piece of the code within the recursion function.  I’m running the first test on my user directory, C:Users
kreider.  The recursive count took about 1.5 seconds to count all the files (27k).  I will need to add logic because the file count doesn’t actually attempt to open and create a hash like my hash function does;  so 27k files may actually end up only being 22k or whatever.

Just a file count of C:\users\rkreider (SSD) took about 1.5 seconds for 26k files.

File count of my user directory (SSD disk), no file hashing or other processing done.


Hashing Test Run 1
On this pass, I decided to run the hash on the files.  It took considerably longer, just under 5 minutes.

File hashing recursively of my user directory (SSD).


Something important to note.  Not all 26,683 of the original files scanned were actually hashed for various reasons such as Access Permissions, file already opened by something, etc.
For comparison, the database (SQLite) created 26,505 records and is 5.4MB in size.
Hashing Test Run 2
I moved the file counter further into the hash loop and only increment the counter when a file is successfully hashed.  Here are my results now.

Recursive hash of my user directory (SSD) with a found/processed indicator now.


As you can see, it found 26,684 file and could only process (hash) 26,510.

Comparing the result in GUI to the database with SELECT COUNT(*) FROM file, it matches properly.  The database size remains about the same, 5.39MB.

One thing that I’m trying to decide is whether or not to put some type of progress identifier on the interface.
The thing is, this adds overhead because I have to first get a count of files and that will take x seconds.  In the case of the NAS scan, it took 500+ seconds, over 5 minutes.  So I’d be waiting 5 minutes JUST for a count and then I’d start the file hashing which will take time.  I just don’t know if it is worth it, but it sure would be nice I believe.

Database Schema

CREATE TABLE [file] (
[id] INTEGER  PRIMARY KEY AUTOINCREMENT NOT NULL,
[hash] text  NULL,
[fullname] text  NULL,
[shortname] text  NULL,
[extension] text  NULL,
[mimetype] text  NULL,
[size] intEGER  NULL,
[modified] TIMESTAMP  NULL
);

Office.com Online using WinForms

This isn’t true API access, just a WebBrowser control. I’m looking into the API though which would require me to register my Application even though it’s not a Windows App.

It has a launcher (I haven’t decided exactly how I’ll integrate this) that will launch the Microsoft Office Online services at the click of a button.

As you can see, you can still access the Navigation provided by Microsoft. I fixed the Winform so it doesn’t launch a new window and keeps everything inside the Winform on the WebBrowser control.

I guess I see this side project being useful for someone who wants to use the free Office.com Online services with their Microsoft account and don’t want the full browser experience.


2014-08-08_141453
2014-08-08_141506
2014-08-08_141519
2014-08-08_141525
Some notes to get this to render properly in the WebBrowser control.

I had to force a User Agent to indicate I was using IE11. I think what I was seeing was that the WebBrowser control was defaulting to MSIE7. Made things look like crap.
VB.NET Code for this:

Private Property uag As String = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko"
     _
    Private Shared Function UrlMkSetSessionOption(ByVal dwOption As Integer, ByVal pBuffer As String, ByVal dwBufferLength As Integer, ByVal dwReserved As Integer) As Integer
    End Function
    Const URLMON_OPTION_USERAGENT As Integer = &H10000001
    Public Function ChangeUserAgent(ByVal Agent As String)
        UrlMkSetSessionOption(URLMON_OPTION_USERAGENT, Agent, Agent.Length, 0)
    End Function

I implemented forcing new windows to the WebBrowser control in the Winform. If you use the top navigation at all, it opened the service in a new Internet Explorer Window.
VB.NET Code for this:

    Private Sub Webbrowser1_NewWindow(sender As Object, e As CancelEventArgs) Handles WebBrowser1.NewWindow
        WebBrowser1.Navigate(WebBrowser1.StatusText)
        e.Cancel = True
    End Sub

Originally, I wanted to just wrap this in a HTA or something simple… but I got errors indicated that I couldn’t pull this in an iFrame. So I tried some Ajax/jQuery stuff in HTA and that was a complete fail (I’m not familiar with ajax/jquery things).

Change User-Agent in WebBrowser Control

This snippet of code worked when tested in VB.NET application I’m working on. It will set the User Agent string for the duration of the control.

Private Property uag As String = Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko
     _
    Private Shared Function UrlMkSetSessionOption(ByVal dwOption As Integer, ByVal pBuffer As String, ByVal dwBufferLength As Integer, ByVal dwReserved As Integer) As Integer
    End Function
    Const URLMON_OPTION_USERAGENT As Integer = &H10000001
    Public Function ChangeUserAgent(ByVal Agent As String)
        UrlMkSetSessionOption(URLMON_OPTION_USERAGENT, Agent, Agent.Length, 0)
    End Function