Author: Rich

Just another IT guy.

Deleting duplicate contacts in MDaemon address book

Somehow the contacts for an email account amassed almost 7k entries. Upon manual review, it seemed there were at least 3 entries for each contact. Here’s a powershell script to create a new MDaemon AddrBook.mrk file with removed duplicates. The script as written compares if the first name, last name, and full name are concatenated together into a single key. This means it requires all three fields to match to consider a contact as a duplicate. It isn’t 100% but it helped significantly reduce the duplicates to a manageable number for reviewing manually.

Here’s an abbreviated format of the AddrBook.mrk file:

<?xml version="1.0" encoding="UTF-8"?>  
<addressBook version="9.5.5" encoding="utf-8" lastModified="2024-07-08T18:05:22.885Z" fid="{1d30e328-d4f6-4e98-bd22-c99fd9523ab3}" NextID="1">  
  <contact>  
    <guid><![CDATA[73e3293156c74f8195dac9d1625645e2]]></guid>  
    <modified>2022-06-13 13:59:33</modified>  
    <firstName><![CDATA[John]]></firstName>  
    <lastName><![CDATA[Doe]]></lastName>  
    <fullName><![CDATA[John Doe]]></fullName>  
    <busCompany><![CDATA[Acme Widgets]]></busCompany>  
    <homeMobile><![CDATA[+1 (555) 555-5555]]></homeMobile>  
    <comment><![CDATA[  
]]></comment>  
  </contact>  
  <contact>  
    <guid><![CDATA[b17e84fa82562e63eef1623ed8475063]]></guid>  
    <modified>2022-06-13 13:59:33</modified>  
    <firstName><![CDATA[Jane]]></firstName>  
    <lastName><![CDATA[Doe]]></lastName>  
    <fullName><![CDATA[Jane Doe]]></fullName>  
    <homeMobile><![CDATA[(555) 555-5555]]></homeMobile>  
    <comment><![CDATA[  
]]></comment>  
  </contact>  
</addressBook>

Here’s the powershell code to extract non duplicate entries into a new file:

# Load the XML file from MDaemon (\users\domain.tld\username\Contacts.IMAP\AddrBook.mrk)  
[xml]$xml = Get-Content -Path "addrbook.mrk"  
  
# Create a hash table to store unique contacts  
$uniqueContacts = @{}  
  
# Function to generate a unique key for each contact  
function Get-ContactKey {  
    param (  
        $firstName,  
        $lastName,  
        $fullName  
    )  
    return "$firstName|$lastName|$fullName"  
}  
  
# Access the contact nodes directly  
$contacts = $xml.addressBook.contact  
  
# Get the total number of contacts before removing duplicates  
$totalContactsBefore = $contacts.Count  
  
# Create a list to store the contacts to remove  
$contactsToRemove = @()  
  
# Iterate through each contact and identify duplicates  
foreach ($contact in $contacts) {  
    $firstName = $contact.firstName.'#cdata-section'  
    $lastName = $contact.lastName.'#cdata-section'  
    $fullName = $contact.fullName.'#cdata-section'  
  
    $key = Get-ContactKey -firstName $firstName -lastName $lastName -fullName $fullName  
  
    if (-not $uniqueContacts.ContainsKey($key)) {  
        # Add contact to unique contacts  
        $uniqueContacts[$key] = $contact  
    } else {  
        # Duplicate found, add to list to remove later  
        $contactsToRemove += $contact  
    }  
}  
  
# Remove duplicate contacts  
foreach ($contact in $contactsToRemove) {  
    $contact.ParentNode.RemoveChild($contact) > $null  
}  
  
# Get the total number of contacts after removing duplicates  
$totalContactsAfter = $xml.addressBook.contact.Count  
  
# Save the updated XML file  
$xml.Save("updated_contacts.xml")  
  
# Write the total numbers to the console  
Write-Output "Total contacts before removing duplicates: $totalContactsBefore"  
Write-Output "Total contacts after removing duplicates: $totalContactsAfter"

Total contacts before removing duplicates: 6871
Total contacts after removing duplicates: 2053

Change from a Microsoft account to a local account

Archived from: Wayback Machine

You can use the Settings app to switch your Microsoft account to a local account.

Important: Microsoft recommends using a Microsoft account, not a local account, when signing in to Windows. Using a Microsoft account in Windows allows for seamless integration of Microsoft services, enhanced security, and sync across devices, unlike a local account.

  1. In the Settings app  on your Windows device, select Accounts > Your info,or use the following shortcut: Your info
  2. Select Sign in with a local account instead. You’ll see this link only if you’re using a Microsoft account
  3. Type the username, password, and password hint. The username must be different than any others on the device
  4. Select Next, then select Sign out and finish. Sign back in with your new local account

Warning: when using a local account, it’s a good idea to create a password reset disk. If you don’t do this and then forget your password, you won’t be able to recover it.

Cleaning up my photo archives

I have taken a lot of photos over the years and things have just piled up and I’ve amassed about 2TB worth of images.

In an effort to get organized, I decided the first step is to reduce the number of images by finding duplicates and also getting rid of the RAW files (CR2, DNG, TIF).

Step One: Removing RAW files

I’ve created a script that will recursively process files in my photo storage directory and convert RAW files to JPEG and then go through and remove the RAW files. I was able to free up about 500GB of storage space by removing the RAW files.

Mogrify is required and is what I use to convert and preserve time stamp as I will later organize photos by date/time.

Convert

@echo off

call :treeProcess
goto :eof

:treeProcess
echo "%CD%"
for %%f in (*.cr2) do (
if not exist "%%~nf.jpeg" (
echo %%f
mogrify -format jpeg -quality 75 -define preserve-timestamp=true "%%f"
)
)
for /D %%d in (*) do (
    cd %%d
    call :treeProcess
    cd ..
)

Cleanup

@echo off

call :treeProcess
goto :eof

:treeProcess
echo "%CD%"
for %%f in (*.cr2) do (
if exist "%%~nf.jpeg" (
echo deleting "%%f"
del /s /f "%%f"
) else (
echo WARN found RAW but no converted JPEG "%%f"
)
)
for /D %%d in (*) do (
    cd %%d
    call :treeProcess
    cd ..
)

I did these for my DNG and TIF files also.

Step Two: Cleaning up Duplicates

For this task, I found a tool called czkawkas and wrote a little about it here. It works well. This process freed up an additional ~10GB. So far, about 512GB of removed unnecessary images.

Giving immich a go on about 200k photos

In the never ending quest to find a photo gallery software to access my photo archive (and possibly synchronize my iPhone to it), I’m now testing Immich in a docker on Windows 11. I’m using an external library instead of uploading the images, which is an 8TB standard disk, so I expect it to be a bit slow. Docker is running, and its storage for thumbnails, on an SSD.

Update 6/20/24

The first phase of ingesting the external library is complete, the next phase will be all the smartphone photos which is on different media. Interesting on the external library that the storage space isn’t showing.