Reclaiming space from thin provisioned disks in VMware ESXi

This is the procedure that has worked for me. There are various methods dating as far back as 2011 across the internet, but this step-by-step works in my environment for Windows and Linux virtual machines.

Step 1 – Prepare the Guest Operating Systems

Windows

  • Download sdelete from Sysinternals (https://live.sysinternals.com/sdelete.exe)
  • In an elevated command prompt, run the following against the volume (in this example, C drive)
    • sdelete -z c:
  • Go to Step 2

Linux

  • For each volume you’ll want to run the following
    • dd if=/dev/zero of=/volume/zero.fill bs=1024k
  • After that completes, remove the zero.fill file from each volume.
    • rm /volume/zero.fill
  • To to Step 2

Step 2 – Punch some Holes

  • Shut down the virtual machine and power it off in vCenter or on the ESXi host
  • Log into the ESXi host via SSH
  • Navigate to the volume where the VM is stored
    • cd /vmfs/volumes/datastore/VirtualMachine
  • Punch the holes
    • vmkfstools -K VirtualMachine.vmdk
  • When completed, back in vCenter or the ESXi host web management, remove the VM from inventory (do not delete from disk). Use the following commands as root on the ESXi host also.
    • Unregister VM (More Info)
      • vim-cmd vmsvc/unregistervm vmid
        • You can get the vmid using vim-cmd vmsvc/getallvms
  • Add the VM back into inventory (go to the datastore and browse for where it is at; select the .vmx and click Register VM). Use the following commands as root on the ESXi host also.
    • Register VM (More Info)
      • vim-cmd solo/registervm /vmfs/volumes/datastore_name/VM_directory/VM_name.vmx

Initialization of Client failed: Closing Application

After installing Rand McNally IntelliRoute software, users may experience the error “Initialization of Client failed: Closing Application”

This problem is usually attributed to inadequate permissions for the user running the application. Please ensure that the folder that Rand McNally software was installed (generally C:\Program Files (x86)\Rand McNally) has Full Control access granted to users. Also, the Registry Key HKEY_LOCAL_MACHINE\SOFTWARE\Rand McNally on the workstation needs to grant “Full Control” to users who execute the program at the desktop level.

32bit Installation Error

The above issue had to do with 64bit installation of IntelliRoute. Below is the issue I experienced with the 32bit installation of IntelliRoute.

I installed the application as an administrator and changed permissions on the Registry Key as well as the Program Files directory as noted above. However, I still received the error.

Using procmon, I tried to get some clues as to why it was crashing. I saw a hint that it looked for a file in C:\Users\%username%\WINDOWS\Sysupz.com.

Upon further investigation, this file did not exist for my test user, however it did exist in for the administrative user that I installed the application under.

To test it out, I copied the file to my test user WINDOWS folder and attempted to launch IntelliRoute and it was a success.

To copy the file to all the users on the server, I used the following command:

cd c:\users\
for /d %u in ("*") do copy c:\users\testuser\WINDOWS\Sysupz.com c:\users\%u\WINDOWS\

This seemed to resolve the issue. I’m not certain if the problems with installation are due to it being installed on a Server 2008 terminal server. That system is slated for migration to Server 2019 in December, so hopefully the IntelliRoute software will not give me problems then!

GoFileRoom Error: The service you are trying to reach is unavailable. Please try again later.

TL;DR

GoFileRoom made changes to their encryption by using TLS1.2. In order for GFR add-on to work in Windows Server, you must modify 2 registry entries for .NET enforcing strong encryption. For GFR website to operate, you need to ensure TLS1.2 is enabled as well.
Here’s the GoFileRoom technical notes on enabling this via Registry: http://cs.thomsonreuters.com/ua/gfr/digita_uk_en/kb/recommended-registry-changes-for-tls-1-2.htm?mybanner=1

How I got here…

Let me preface this with the fact that GoFileRoom is not officially supported on Server 2016, so my troubleshooting process was skewed because I was thinking there was an installation problem or some other incompatibility issue.

The GFR Windows add-on recently stopped working as reported by users on a terminal server.  The error being thrown was right at the logon prompt of the GFR Add-on with the following message:

The service you are trying to reach is unavailable. Please try again later.
width=479

Digging around the system, I managed to find a logfile that GFR saves to, it is located in %APPDATA%\GoFileRoom\GFRControlPanel\logs

The log file didn’t give me a single hint:

[GoFileRoom.GFRUser]:[Login] Web Exception: The request was aborted: Could not create SSL/TLS secure channel.

When in doubt… procmon!

This didn’t directly answer any of my questions as to why… so, when in doubt, procmon!  I went and grabbed ProcMon from SysInternals and ran it during a GFR session so I could figure out if there was some incompatibility.
ProcMon didn’t give me much of anything.  I saw it was attempting an HTTPS connection to member.gofileroom.com, but nothing abnormal in terms of compatibility problems or something.

I went back to the GFR logfile message and did a quick Google for The request was aborted: Could not create SSL/TLS secure channel.
The first result was a StackOverflow post and it was a great post with two points of interest to me:

  1. Mentions of TLS 1.2 via HTTPWebRequest method
  2. SChannel logging

This got the wheels in my head turning — There was one specific comment that said to look for a TLS 1.2 support issue and another comment about enabling logging for SChannel.

I knew IE was configured with TLS properly as that was one of the first things I checked (per GFR installation guidelines).  But there’s something different here.  It seems the GFR add-on is calling an HTTPWebRequest method and that is where this error message is being generated from!

Further into the thread of comments and proposed solutions, there was a note about enabling SChannel – totally forgot about this as I wasn’t thinking it was a TLS negotiation issue!  I enabled SChannel logging at level 7 (all messages) and reproduced the error.  Fired up event viewer and saw what I needed:

A fatal alert was received from the remote endpoint. The TLS protocol defined fatal alert code is 40.


width=548

No idea what code 40 is, so off to MSDN.

An MSDN article tells me it is a handshake_failure.  Awesome, now I have a better search term to try for Google: gofileroom tls

Guess what?  First result is a GoFileRoom KB article.  See the TL;DR for the link.
#net-framework, #gfr, #gofileroom, #tls

CPU-miner Installed via Windows OS Vulnerability

Update 5/6/2017:  Close port 445 and apply MS 17-010
I have triaged a handful of Windows servers this week that started out being ticketed as high CPU / performance issues.
Upon investigation, I have found XMR cryptocurrency miners being installed through a Windows OS Vulnerability.
Continue reading CPU-miner Installed via Windows OS Vulnerability

System and Compressed Memory

I was looking at my system’s performance (Windows 10 Build 10565) and noticed something I hadn’t seen before “System and compressed memory” utilizing about 500MB. Looking into this further, an explanation from Microsoft is below on this new concept added in Build 10525.


width=638

Memory Manager Improvements:

In Windows 10, we have added a new concept in the Memory Manager called a compression store, which is an in-memory collection of compressed pages. This means that when Memory Manager feels memory pressure, it will compress unused pages instead of writing them to disk. This reduces the amount of memory used per process, allowing Windows 10 to maintain more applications in physical memory at a time. This also helps provide better responsiveness across Windows 10. The compression store lives in the System process’s working set. Since the system process holds the store in memory, its working set grows larger exactly when memory is being made available for other processes. This is visible in Task Manager and the reason the System process appears to be consuming more memory than previous releases.

Reference(s):

  1.  http://blogs.windows.com/windowsexperience/2015/08/18/announcing-windows-10-insider-preview-build-10525/
  2. http://superuser.com/questions/952141/windows-10-system-process-taking-massive-amounts-of-ram
  3. https://channel9.msdn.com/Blogs/Seth-Juarez/Memory-Compression-in-Windows-10-RTM
  4. http://forum.sysinternals.com/need-help-with-ntoskrnl-thread-causing-high-cpu_topic29289_post146738.html#146738