apoc/ Documentation/ TroubleShooting

Troubleshooting APOC

This page describes commonly encountered problems using the Configuration Manager framework and their possible causes, as well as steps to test these causes and hopefully fix the problems.

  1. Troubleshooting APOC

APOC Manager/Web Console Issues

Cannot Install

  • Symptom: at the end of the installation of the Web Console, a message states that it won't start because there are no registered applications.
  • Possible causes:
    • this is perfectly normal, once the APOC Manager module is installed it will start the Web Console.

Connection Refused

  • Symptom: you've pointed your browser to the appropriate URL, i.e https://<your.server>:6789, but it tells you the connection is refused.
  • Possible causes:
    • the Web Console is not running on the server.

Cannot Login

  • Symptom: you've reached the login page of the Web Console, but the user/password combination is rejected.
  • Possible causes:
    • the LDAP server is not running.
    • the Web Console LDAP authentication module is misconfigured.
    • the user doesn't exist in the LDAP server.
    • the user has a different password in the LDAP server.
  • Symptom: you've logged in the Web Console, but the application list page doesn't display the APOC Manager.
  • Possible causes:
    • the APOC Manager module is not installed.
    • the user doesn't exist in the LDAP server but does exist in the users map on the machine.

Null Pointer Exception, Tomcat/Java Error or Blank Page

  • Symptom: you've entered the APOC Manager but nothing of value is displayed, just a blank page or some errors.
  • Possible causes:
    • if the error mentions NoClassDefFoundError: sun/tools/javac/Main, the Web Console uses the wrong Java.
    • the APOC Manager module is misconfigured.
    • the LDAP server has been incorrectly prepared.

Desktop Issues

Cannot See Any Central Settings At All

  • Symptom: no matter which setting is valued using the APOC Manager, no matter for which application (Gnome, Mozilla, ?StarOffice, Evolution), the central values are not visible on the desktop.
  • Possible causes:
    • the APOC Agent is not enabled.
    • the LDAP server is not running.
    • the LDAP server has been incorrectly prepared.
    • the current user doesn't exist in the LDAP server.

Cannot See Any Gnome or Evolution Settings

  • Symptom: ?StarOffice and Mozilla work fine, but the central settings for Gnome applications, including Evolution are not taken into account.
  • Possible causes:
    • the GConf adapter is not installed or not setup properly.

Cannot See Any Mozilla Settings

Cannot See Any StarOffice Settings

Cannot See Some Settings

  • Symptom: some settings set using the APOC Manager are not taken into account, while others for the same application are fine.
  • Possible causes:
    • a local user preference has been defined for that setting.

Settings Are Not Refreshed

  • Symptom: the value of a setting has been changed using the APOC Manager and the new value is not propagated to the desktop application.
  • Possible causes:
    • a local user preference with the same value as the old value has been defined for that setting.
    • the APOC Agent change detection cycle has not yet taken place.

Staroffice And Mozilla Crash On Linux Sun Ray

  • Symptom: ?StarOffice crashes ( crash reporter displayed ) and Mozilla crashes ( appears not to start ). Look for ordinal="os_exists+0x50" near the top of the ?StarOffice crash reporter stack.
  • Possible causes:

    • incompatibility between /usr/lib/libc ut.so ( provided by Sun Ray to address audio problems with Sun Ray devices ) and BerkeleyDB used by apoc.

Tests & Solutions

LDAP Server

The instructions below are in some cases specific to the Sun Directory Server, especially regarding process names and default file locations.

Starting the Server

To check that the LDAP server is running, run the command ps -ef | grep slapd. If no process is listed, the LDAP server is not running. To remedy this, run the command /var/Sun/mps/slapd-<instance>/start-slapd, replacing with the instance name of your LDAP server (there will anyway be only one directory with that naming structure in /var/Sun/mps).

Preparing the Server for APOC

To ensure that the LDAP server has been properly setup, perform the following command: 

ldapsearch [-x] -h <host> -p <port> -b ou=apocregistry,ou=default,ou=organizationconfig,ou=1.0,ou=apocservice,ou=services<base> ou=apocregistry sunkeyvalue`

where , , and match the values found in either the APOC Manager configuration or the APOC Agent configuration. The result of this search should be one entry with unreadable attributes (base 64 encoded values). The -x flag may be required or not depending on the version of ldapsearch used, try with it and remove it if a usage message is displayed. If no entry is found, that particular on that particular has not been prepared for use with APOC.

Follow the instructions in the installation guide to run the deployment tool and prepare the LDAP server, please note that if the LDAP server uses non-standard attributes or object classes for its organisations, roles or users, or if users are not stored in ou=People containers, the file named ?OrganisationMapping needs to be modified to reflect these differences prior to running the deployApoc or createServiceTree scripts.

Finding a User in LDAP

To check whether a given user identified by its exists in the LDAP server, run the following command from any machine:

ldapsearch [-x] -h <host> -p <port> -b <base> <attribute>=<username> dn

where , , and match the values found in either the APOC Manager configuration or the APOC Agent configuration, and is the unique identifier attribute specified during the Web Console configuration or found in the organisational mapping file. The -x flag may be required or not depending on the version of ldapsearch used, try with it and remove it if a usage message is displayed.

If the above command returns Cannot connect, the LDAP server is not running. If it returns no entries, the user doesn't exist in the server or the attribute used is wrong. Add the user to the server and/or verify the APOC Manager configuration and the organisational mapping file. If it returns multiple entries, this means the attribute used doesn't uniquely identify a user in the LDAP server and this must be changed, either by using a different attribute or by modifying one of the duplicate entries.

Testing a User Password in LDAP

To check that a user/password combination is valid for use with the APOC Manager, first run the command used to find that user. The result of that command will contain a line formatted like dn: uid=username,<blablabla>,<base>. The string after dn: is the full DN of the user. Then perform the same search, but this time authenticated as that user, by adding before the -h flag in the command the string -D "<full DN>" -w <password>. If the command returns Invalid credentials, the user/password combination is invalid. The best course of action is then to reset the password to a known value using an LDAP administrative tool.

Web Console

Starting the Web Console

To check whether the web console is running, run the command smcwebserver status. If it's not running, it can be started with the command smcwebserver start.

Configuring the Web Console

To check the web console LDAP authentication module configuration, run the command smreg list -m on the web console server. The output should mention a module named com.sun.apoc.authentication.LdapLoginModule and list its parameters. Check that the server name and port number, as shown in the ldap.provider.url parameter are pointing to your LDAP server, that the base DN from the ldap.baseDN parameter is the top of your LDAP data tree, and that the LDAP attribute listed in the ldap.userAttribute is indeed the attribute used to uniquely identify user entries in the LDAP server.

If the configuration is incorrect, it can be rectified by going to the directory /usr/share/webconsole/apoc and running the command ./configure. The tool will prompt for the new configuration parameters, please note the following points:

  • LDAP server name: the host name or IP provided here must be reachable by that name from the web console machine;
  • LDAP search operations user DN and password: these fields can be left blank if the LDAP server allows anonymous connections and searches (which is the case by default for Sun Directory Server and OpenLDAP). If anonymous access is disabled, the user DN provided must be the full DN of an entry in the LDAP server which has enough permissions to do read and search operations. It doesn't need to be able to perform write operations, and as such, using here the full DN of an administrator entry is overkill.

    Setting the Web Console Java Environment

The current Web Console Java environment can be checked by running smreg list -p and looking at the java.home property. This property must point to a valid Java home, and furthermore that Java home must be a JDK and not just a JRE. If this value is incorrectly set, it can be modified by running smreg add -p java.home=<JAVA_HOME>, with pointing to a valid installation, i.e where the executables javac can be found in the bin/ subdirectory.

APOC Manager

Installing APOC Manager

To check if the APOC Manager is installed in the Web Console, run smreg list -a. One of the registered applications must be com.sun.apoc.manager_. If such is not the case, follow the instructions in the installation guide to install the module.

Configuring the APOC Manager

To check if the APOC Manager is pointing to the proper LDAP server, check the contents of the file named policymgr.cfg located in /usr/share/webconsole/apoc/WEB-INF. That file should contain the host name, port number and base DN of the LDAP repository. If the data in that file is not correct, use the configure script as described in the second half of the section on configuring the Web Console to provide the proper values. Checking the values are correct can be done by trying to find a user in the directory using these parameters.

Enabling Debug Log Level for the APOC Manager

The current log level of the APOC Manager, which is actually the log level of the Web Console, can be checked by running smreg list -p and looking at the debug.trace.level property. To force the APOC Manager to generate more traces so that the source of problems can be investigated, this property must be changed by running smreg add -p debug.trace.level=<log level>, where is 3 for the maximum amount of traces.

APOC Agent

Enabling the APOC Agent

To enable the APOC Agent you need to configure the policymgr.properties, file located in /usr/local/etc by default, here's an example of a basic anonymous configuration: 

#-----------------------------#
# Basic Configuration Example #
#-----------------------------#

#PROVIDER_URL = ldap://server1.freedesktop.org:389/o=mySuffixDn
#AuthType=Anonymous
#AuthDn=
#Password=
#ConnectTimeout=1

Configuring the APOC Agent

The configuration of the APOC agent is done running the apoc-config tool.

It is important that the details of the LDAP server, specifically the host name, port number and base DN accurately allow the agent to communicate with the server. One good way of ensuring this is by using the values in these tools to find a user in the LDAP repository.

Enabling Debug Log Level for the APOC Agent

To get the APOC agent to generate as many traces as possible to facilitate the investigation of problems, its log level must be changed to the value FINEST. This can be done editing the file named apocd.properties located in /etc/apoc, if a line containing LogLevel exists in that file, replace that line with the exact string LogLevel=FINEST or if no such line exists, just add the string at the end of the file. Force the agent to reload its configuration by running the command /usr/lib/apoc/apocd reload.

Managing Change Detection in the APOC Agent

By default, the APOC agent checks for changes to the configuration of currently logged in users every hour. It can be forced to perform a change detection by running /usr/lib/apoc/apocd change-detect. To modify the change detection frequency use the agent configuration tool. Another way is to edit the file apocd.properties, located in /etc/apoc and, if a line containing ChangeDetectionInterval (not DaemonChangeDetectionInterval) exists, replace it with the string ChangeDetectionInterval=<period>, where <period> is the number of minutes between detection cycles, or if no such line exists, just add the string at the end of the file. Force the agent to reload its configuration by running the command /usr/lib/apoc/apocd reload.

GConf Daemon

Enabling APOC in GConf

To check whether APOC is enabled in GConf, verify that the file path located in /etc/gconf/2 contains lines like apoc:readonly:mandatory@ and apoc:readonly and that these lines are not commented out. The fact that a proper session with the APOC agent has been established when a GConf daemon has started can be tested on Linux by looking in the file messages located in /var/log and looking for gconfd messages associated with the user.

Desktop User Preferences

If a user has explicitly chosen preferences for the desktop applications, usually by going to the Preferences menu, these preferences will be used by the application instead of the central values set in the APOC Manager. To force a central value to be used, i.e to prevent users from modifying it, the value must be protected. Refer to the administration guide to learn how to protect a value.

Incompatibility between Sun Ray /usr/lib/libc_ut.so and BerkeleyDB

We are currently working with the Sun Ray team to figure out exactly why this incompatibility exists. As a temporary workaround we are providing an alternative version of libc_ut.so available here for download. Once downloaded, you can simply copy this file to /usr/lib/libc_ut.so on your Sun Ray server and reboot.

Please note that, until we have come to an agreement with the SunRay team on how to correctly address this problem, we cannot vouch for the correctness of this workaround.

When All Else Fails

If the problem doesn't fit one of the above issues, or the tests and solutions offered failed to remedy it, contact the mailing list, providing a description of the problem and if possible root access to the machine affected by the problem (either the Web Console machine or a desktop one). If providing a root access is not possible, enable debug information on the appropriate machine (the Web Console or the desktop), recreate the problem and depending on the nature of the problem, gather and make available (by attaching to the mail or otherwise) the following information:

  • if everything seems to be started and running, but no access to the LDAP server, i.e login or data retrieval, can be achieved, get from the LDAP server:
    • a copy of the log files located on a Sun Directory Server in /var/Sun/mps/slapd-<instance>/logs and named access and errors, making sure that the logs cover the time at which the problem occurred (these log files are rotated when they reach a certain size, the old files are named with timestamps indicated the period they cover).
    • the output of ps -ef.
    • if possible, an LDIF dump of the contents of the directory.
  • if the problem is happening in the APOC Manager, get from the Web Console machine:
    • the output of ps -ef.
    • the output of smreg list.
    • the output of smcwebserver status.
    • for a RPM based Linux machine, the output of rpm -qi SUNWmcon and rpm -q apoc-manager, for a Solaris one the output of pkginfo -l SUNWmcon and pkginfo -l SUNWapm.
    • a copy of the file named policymgr.cfg located in /usr/share/webconsole/apoc/WEB-INF.
    • a copy of the file named consoledebuglog located in /var/opt/webconsole.
  • if the problem is happening on a desktop, get from the desktop machine:
    • the output of ps -ef.
    • a copy of the file named policymgr.properties located in /opt/apoc/lib for JDS 2 or /etc/apoc for JDS 3.
    • a copy of the file named apocd.log.0 located in /var/opt/apoc/Logs.
    • on Linux only, a copy of the file named messages located in /var/log.
    • for a RPM bassed Linux machine, the output of rpm -qa | grep apoc, for a Solaris one the output of pkginfo | grep SUNWap.