Apache Segfault when Navigating to WP-Admin

width=588

Apache

Argh!

When trying to get into WordPress’ administration panel, Apache segfaults.

First troubleshooting steps involved moving all plugins/ to a temporary folder.

This didn’t produce any new results, still segfaulting.

Here are some notes on how I went about debugging this with GDB 7.3

wp-admin crashing

setup core dump in /etc/apache2.conf

CoreDumpDirectory /tmp-apache2/

Restart apache.

/etc/init.d/apache2 restart

install GDB 7.1+ for PIE support (See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=346409)
gdb Download: http://www.gnu.org/software/gdb/download/

root@nitrous:/tmp-apache2# wget -d http://ftp.gnu.org/gnu/gdb/gdb-7.3a.tar.bz2
root@nitrous:/tmp-apache2# bzip2 -d gdb-7.3a.tar.bz2
root@nitrous:/tmp-apache2# tar xvf gdb-7.3a.tar
root@nitrous:/tmp-apache2# cd gdb-7.3/
root@nitrous:/tmp-apache2# apt-get install gcc make ncurses-dev apache2-dbg php5-dbg
root@nitrous:/tmp-apache2/gdb-7.3# ./configure && make && make install

navigate to page, wp-admin to create crash

/usr/local/bin/gdb /usr/sbin/apache2 /tmp-apache2/core

(gdb) thread apply all bt full

root@nitrous:/tmp-apache# /usr/local/bin/gdb `which apache2` core
GNU gdb (GDB) 7.3
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/apache2...(no debugging symbols found)...done.
[New LWP 5596]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f28ad87f5f9 in _zend_hash_add_or_update () from /usr/lib/apache2/modules/libphp5.so
(gdb) bt
#0 0x00007f28ad87f5f9 in _zend_hash_add_or_update () from /usr/lib/apache2/modules/libphp5.so
#1 0x00007f28ad82cae5 in php_register_variable_ex () from /usr/lib/apache2/modules/libphp5.so
#2 0x00007f28ad6a21a9 in ?? () from /usr/lib/apache2/modules/libphp5.so
#3 0x00007f28ad826542 in sapi_getenv () from /usr/lib/apache2/modules/libphp5.so
#4 0x00007f28a3d895d4 in suhosin_log () from /usr/lib/php5/20090626/suhosin.so
#5 0x00007f28a3d84b3e in ?? () from /usr/lib/php5/20090626/suhosin.so
#6 0x00007f28ad8858a5 in ?? () from /usr/lib/apache2/modules/libphp5.so
#7 0x00007f28ad885981 in ?? () from /usr/lib/apache2/modules/libphp5.so
#8 0x00007f28ad87d885 in zend_hash_apply () from /usr/lib/apache2/modules/libphp5.so
#9 0x00007f28ad885df0 in zend_ini_deactivate () from /usr/lib/apache2/modules/libphp5.so
#10 0x00007f28ad87132f in ?? () from /usr/lib/apache2/modules/libphp5.so
#11 0x00007f28ad81cb65 in php_request_shutdown () from /usr/lib/apache2/modules/libphp5.so
#12 0x00007f28ad8fe177 in ?? () from /usr/lib/apache2/modules/libphp5.so
#13 0x00007f28b16a2880 in ap_run_handler ()
#14 0x00007f28b16a621e in ap_invoke_handler ()
#15 0x00007f28b16b3e48 in ap_process_request ()
#16 0x00007f28b16b0cf8 in ?? ()
#17 0x00007f28b16aa478 in ap_run_process_connection ()
#18 0x00007f28b16b8e97 in ?? ()
#19 0x00007f28b16b91aa in ?? ()
#20 0x00007f28b16b94db in ap_mpm_run ()
#21 0x00007f28b168ea90 in main ()
(gdb) bt full
#0 0x00007f28ad868f84 in zend_cleanup_function_data_full () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#1 0x00007f28ad87d885 in zend_hash_apply () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#2 0x00007f28ad868f50 in zend_cleanup_class_data () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#3 0x00007f28ad87d885 in zend_hash_apply () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#4 0x00007f28ad8648f0 in ?? () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#5 0x00007f28ad8712a2 in ?? () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#6 0x00007f28ad81cb65 in php_request_shutdown () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#7 0x00007f28ad8fe177 in ?? () from /usr/lib/apache2/modules/libphp5.so
No symbol table info available.
#8 0x00007f28b16a2880 in ap_run_handler ()
No symbol table info available.
#9 0x00007f28b16a621e in ap_invoke_handler ()
No symbol table info available.
#10 0x00007f28b16b3c6c in ap_internal_redirect ()
No symbol table info available.
#11 0x00007f28aa9e8cb5 in ?? () from /usr/lib/apache2/modules/mod_rewrite.so
No symbol table info available.
#12 0x00007f28b16a2880 in ap_run_handler ()
No symbol table info available.
#13 0x00007f28b16a621e in ap_invoke_handler ()
No symbol table info available.
#14 0x00007f28b16b3e48 in ap_process_request ()
No symbol table info available.
#15 0x00007f28b16b0cf8 in ?? ()
No symbol table info available.
#16 0x00007f28b16aa478 in ap_run_process_connection ()
No symbol table info available.
#17 0x00007f28b16b8e97 in ?? ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#18 0x00007f28b16b91aa in ?? ()
No symbol table info available.
#19 0x00007f28b16b9e34 in ap_mpm_run ()
No symbol table info available.
#20 0x00007f28b168ea90 in main ()
No symbol table info available.

I have no idea where the problem is still.

Some googling found some hints at suhosin possibly, so I modified /etc/php5/apache/config.ini and commented out the extension=suhosin.so.

Restarted apache

/etc/init.d/apache2 restart

Navigated to wp-admin – everything works!

Now, why/what is causing suhosin to segfault? That is still under investigation.

How to get current MS SQL 2008 Version

Here’s a quick T-SQL to get you the current version of your SQL server.

SELECT
SERVERPROPERTY('ProductVersion') AS ProductVersion,
SERVERPROPERTY ('ProductLevel') AS ProductLevel,
SERVERPROPERTY('Edition') AS Edition,
SERVERPROPERTY('EngineEdition') AS EngineEdition
GO
width=494

Microsoft

You can also use:

SELECT SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition')

XP Windows Task Scheduler Advanced Menu

Windows XP’s Tasks folder view has an Advanced menu (which I’ve never paid attention to) that allows you to choose additional functions for the task scheduler.

I *never* knew this existed via the GUI.  I always used the command-line and registry snooping to find/create tasks and the event viewer to review task’s runs.  This menu lets you show hidden tasks, view the log, setup notification, etc.

Windows Task Scheduler Advanced Menu

Ncurses Motha F##ka

I love when I read through code as I’m hacking it up and I actually laugh out loud.  This one got me today when I was hacking up Linfo (PHP Linux System information script).

// Extensions
runExtensions($info, $settings);

// Make sure we have an array of what not to show
$info['contains'] = array_key_exists('contains', $info) ? (array) $info['contains'] : array();

// From the command prompt? Ncurses motha fucka!
if (defined('LINFO_CLI')) {
        $out = new out_ncurses();
        $out->work($info, $settings, $getter);
}

// Coming from a web server
else {
        // Decide what web format to output in
        switch (array_key_exists('out', $_GET) ? $_GET['out'] : 'html') {

                // Just regular html
                case 'html':
                default:
                        showInfoHTML($info, $settings);
                break;

Ah, makes the day seem not too bad when I see this stuff.

UPHCleanup, Handle, ProcExp vs. NTUSER.DAT

The infamous:

Windows cannot load the user’s profile but has logged you on with the default profile for the system.

This was received on Windows Server 2003 Terminal Server (non-roaming profile) when I logged in this morning.  I *really* grumble when I see these errors.

My first thought was NTUSER.dat and NTUSER.dat.LOG files were locked.

This server has User Profile Hive Cleanup which is supposed to combat this during a user log off and make sure handles are closed to the files in the user’s profile.

First things first, I grabbed Process Explorer from SysInternals website.  I know that in the application  you can search for a file and close it’s handle (from previous experience) so I got to work.

I fired up ProcExp and searched for NTUSER.dat then sorted by handle.

width=597

Process

I double-clicked on NTUSER.DAT and then right-clicked and chose Close Handle.

Ah damn, didn’t work.  (Shocking?  Pfft.)

Next, I decided to grab SysInternal’s handle command-line tool.

C:CMDS>handle -u ntuser.dat|find /i rkreider
System             pid: 4      type: File          NT AUTHORITYSYSTEM       27D4: C:Documents and Settingsrkreiderntuser.dat.LOG
System             pid: 4      type: File          NT AUTHORITYSYSTEM       2B9C: C:Documents and SettingsrkreiderNTUSER.DAT

I need the PID and handle (hex) from the information above (bolded), then I could try to kill it:

C:CMDS>handle -c 27D4 -p 4

Handle v3.46
Copyright (C) 1997-2011 Mark Russinovich
Sysinternals - www.sysinternals.com

 27D4: File  (---)   C:Documents and Settingsrkreiderntuser.dat.LOG
Close handle 27D4 in System (PID 4)? (y/n) y

Handle closed.

C:CMDS>handle -c 2b9c -p 4

Handle v3.46
Copyright (C) 1997-2011 Mark Russinovich
Sysinternals - www.sysinternals.com

 2B9C: File  (---)   C:Documents and SettingsrkreiderNTUSER.DAT
Close handle 2B9C in System (PID 4)? (y/n) y

Handle closed.

Now, you’d think it would be closed, right since ‘handle’ told me so.  Let’s check!

C:CMDS>handle -u ntuser.dat|find /i rkreider
System             pid: 4      type: File          NT AUTHORITYSYSTEM       27D4: C:Documents and Settingsrkreiderntuser.dat.LOG
System             pid: 4      type: File          NT AUTHORITYSYSTEM       2B9C: C:Documents and SettingsrkreiderNTUSER.DAT

Aaaiiieee! It didn’t work. =(

I decided to make sure UPHClean.exe was running (User Profile Hive Cleanup) with a quick

tasklist | find /i uphclean.exe

It is running I saw so I decided for S&G (shits & giggles) to just restart the service.

After I restarted the service, I once again used handle to quickly locate the files using:

handle -u ntuser.dat | find /I rkreider

BINGO!

It was gone.  So, what is the root problem here?  UPHClean not doing it’s job?  *shrug*