Case of Dead Path on ESXi

I had 8 paths go down to a dead state on an ESXi host.  The paths were MRU via Fiber Channel to a storage array.  One path worked and it was configured as RR path.

I knew this wasn’t a physical issue, it had to be a software/configuration issue on my host because there were:

  • No storage array errors
  • Additional hosts in the cluster had no problems
  • One path still worked from the HBA

Looking at the log (/var/log/vmkernel.log) I searched for one of the LUN identifiers, in my case “:L30” which was one of the dead paths.  This yielded a result showing an error with NMP plugin driver invalid command.

Next step was to figure out and verify what NMP details were and compare against a working host.

esxcli storage nmp device list |grep "Path Selection Policy:" |sort |uniq -c

I saw nothing out of the ordinary.

Apparently the storage did not like the use of RR so I removed the SATP claim rule though, so I removed it:

esxcli storage nmp satp rule remove -V IBM -M "^1746*" -P VMW_PSP_RR -s VMW_SATP_ALUA

Storage paths are happy now.

WP Preserve Backslashes

I created a WordPress plugin based on a personal dilemma I ran into with my site being stripped of backslashes.

Upon post save, it converts backslashes to HTML entity \ which is what will be stored in the database.

The plugin is available on GitHub at https://github.com/rjkreider/wp-preserve-backslashes

Here’s the function if you want to just drop it in your functions.php file instead of installing it as a plugin.

function wppb_keepbackslash($PostID) {
    $thePost = get_post($PostID);
    $Content = str_replace('\\', '\', $thePost->post_content);
    // unhook this function so it doesn't loop infinitely
    remove_action( 'save_post', 'wppb_keepbackslash' );
    $UpdatedPost = array (
          'ID'           => $PostID,
          'post_title'   => $thePost->post_title,
          'post_content' => $Content
        );
  wp_update_post( $UpdatedPost );
/** if (is_wp_error($post_id)) {
 $post_id=   wp_update_post( $UpdatedPost );
        $errors = $post_id->get_error_messages();
        foreach ($errors as $error) {
                echo $error;
        }
}   **/
    // re-hook this function
    add_action( 'save_post', 'wppb_keepbackslash' );
}
add_action('save_post', 'wppb_keepbackslash' ); // Update Content when saving content

All my backslashes are gone in WordPress. Yikes.

Discovered that my most recent conversion from SQLite to MySQL seems to have screwed up my backslashes in all my posts that have backslashes.

This is bad because my code snippets should not be copy & pasted and run at face value unless you verify the code!  It could seriously break shit.

Ugh.  This is going to be a PITA to go and fix 500 posts.   It might be quicker to try to fix the SQLite DB file and try another conversion.  This isn’t the first time I’ve noticed this problem.  I see the issue when I restore from XML files as well, and even just copying database using something like mysqldump to dump and then importing using mysql command.  I’m probabaly just missing a simple flag to not strip slashes or something.

My next step is to confirm if there is actually a backslash in the SQL data and it is just being stripped in the_content() or something;  or if the backslash is REALLY not there.  *sad face*

update 1:  the slashes are not in SQL.  Looks like I need to look at my export DB to see if they are in that. *crosses fingers*

update 2:  found this article that creates a function to convert backslashes into HTML entities as the posts are saved.  https://www.tweaking4all.com/web-development/wordpress/preserve-backslash-in-posts/#comment-268277

WordPress SQLite to MySQL Migration Complete

Just finished migrating my website from SQLite to MySQL. What a rush. (lol)

It was actually not as bad as I thought. A lot of sed, grep and other sorcery involved; especially in transforming of SQLite statements to MySQL.

Some quick commands I used:

sqlite techish.db .dump > production_2018-08-23.dump.sql

I found that it used quotes for tables and column names, so I had to remove those first and foremost.

sed -i '/INSERT INTO/,/VALUES (/s/"//g' production_2018-08-23.dump.sql

Next I found that there was an error using mysql -ufoo -p mynewdatabase < production_2018-08-23.dump.sql because the table creations were failing still. So I did a quick fresh install of a vanilla WordPress install, did a dump of the database and just grabbed the table creation parts out:

Dump fresh database:

mysqldump -ufoo -p wordpres > wordpress.sql

Next, I just want table creations...

awk '/CREATE TABLE/, /) ENGINE/' wordpress.sql > create_tables.sql

Next, run create_tables.sql on my new database and then import data.

mysql -ufoo -p mynewdatabase < create_tables.sql

Sweet, that worked and I have a baseline of tables now.

Now importing the data...

[pssst, Rich. Finish me!]

#mysql, #sqlite, #wordpress

GoFileRoom Error: The service you are trying to reach is unavailable. Please try again later.

TL;DR

GoFileRoom made changes to their encryption by using TLS1.2. In order for GFR add-on to work in Windows Server, you must modify 2 registry entries for .NET enforcing strong encryption. For GFR website to operate, you need to ensure TLS1.2 is enabled as well.

Here’s the GoFileRoom technical notes on enabling this via Registry: http://cs.thomsonreuters.com/ua/gfr/digita_uk_en/kb/recommended-registry-changes-for-tls-1-2.htm?mybanner=1

How I got here…

Let me preface this with the fact that GoFileRoom is not officially supported on Server 2016, so my troubleshooting process was skewed because I was thinking there was an installation problem or some other incompatibility issue.

The GFR Windows add-on recently stopped working as reported by users on a terminal server.  The error being thrown was right at the logon prompt of the GFR Add-on with the following message:

The service you are trying to reach is unavailable. Please try again later.

width=479

Digging around the system, I managed to find a logfile that GFR saves to, it is located in %APPDATA%\GoFileRoom\GFRControlPanel\logs

The log file didn’t give me a single hint:

[GoFileRoom.GFRUser]:[Login] Web Exception: The request was aborted: Could not create SSL/TLS secure channel.

When in doubt… procmon!

This didn’t directly answer any of my questions as to why… so, when in doubt, procmon!  I went and grabbed ProcMon from SysInternals and ran it during a GFR session so I could figure out if there was some incompatibility.

ProcMon didn’t give me much of anything.  I saw it was attempting an HTTPS connection to member.gofileroom.com, but nothing abnormal in terms of compatibility problems or something.

I went back to the GFR logfile message and did a quick Google for The request was aborted: Could not create SSL/TLS secure channel.

The first result was a StackOverflow post and it was a great post with two points of interest to me:

  1. Mentions of TLS 1.2 via HTTPWebRequest method
  2. SChannel logging

This got the wheels in my head turning — There was one specific comment that said to look for a TLS 1.2 support issue and another comment about enabling logging for SChannel.

I knew IE was configured with TLS properly as that was one of the first things I checked (per GFR installation guidelines).  But there’s something different here.  It seems the GFR add-on is calling an HTTPWebRequest method and that is where this error message is being generated from!

Further into the thread of comments and proposed solutions, there was a note about enabling SChannel – totally forgot about this as I wasn’t thinking it was a TLS negotiation issue!  I enabled SChannel logging at level 7 (all messages) and reproduced the error.  Fired up event viewer and saw what I needed:

A fatal alert was received from the remote endpoint. The TLS protocol defined fatal alert code is 40.

width=548

No idea what code 40 is, so off to MSDN.

An MSDN article tells me it is a handshake_failure.  Awesome, now I have a better search term to try for Google: gofileroom tls

Guess what?  First result is a GoFileRoom KB article.  See the TL;DR for the link.

#net-framework, #gfr, #gofileroom, #tls