Français · Deutsch · 日本語 Troubleshooting Reflection X 14.x XDMCP Connections to UNIX and Linux Hosts
Technical Note 1229
Last Reviewed 01-Nov-2011
Applies To
Reflection X version 14.x
Summary

This technical note describes how to troubleshoot problems connecting with XDMCP (X Display Manager Control Protocol), such as timeout or "cannot open display" errors. The troubleshooting options in this note apply to XDMCP connections made from Reflection X 14.x to the most common UNIX and Linux hosts.

For information specific to Red Hat Linux, see Technical Note 1680.

XDMCP Connections

The XDMCP protocol is used to communicate between the X Display Manager (XDM) running on a host machine and the X server (Reflection X) running on your workstation (computer). The X Display Manager provides a user friendly interface for starting X clients and opening X windows.

Troubleshooting XDM Connection Problems

There are several reasons why an XDMCP connection might fail. The most common reasons are:

  • The XDM daemon is not running on the host, or the host is unavailable.
  • Port 177 (UDP) or port 6000 (TCP) is blocked.
  • The workstation is not correctly configured in the network DNS table.

To determine the cause of the problem and resolve it, start with section A. XDMCP Error Codes, and proceed through each subsequent section as directed.

Note: The troubleshooting steps in this note help you ensure that the XDM daemon is running and that Reflection X can successfully connect and run sessions using this daemon. An alternate workaround for starting sessions is to manually start X sessions. This procedure is described below under "To manually start an X session without using the XDM Daemon."

A. XDMCP Error Codes

The most common error codes and their definitions are:

  • RX2102 - "Your XDMCP connection timed out - make sure that the hosts on your network are running XDM programs. (RX2102)"

This is a generic XDMCP error code. To determine the cause of the error and resolve it, proceed to section B. Examine the Network Environment.

  • RX2112 - "XDM session to host failed (Session XXX failed for display <fully qualified PC name>:0: Cannot open display). (RX2112)"

XDM is running on the host and has accepted the connection request; however, XDM is unable to successfully reply to the Reflection X server. Refer to sections F. Verify that Reflection X is Detecting the Correct IP Address and G. Ping the Workstation from the Host.

B. Examine the Network Environment

Before you begin the troubleshooting steps, be sure that your network environment is compatible with XDMCP. XDMCP does not work with, or needs to be specifically configured for, the following network environments:

  • Secure Shell (SSH): XDMCP connections cannot be created through an SSH tunnel, because SSH does not support UDP transmissions. See Technical Note 1818 for more information.
  • Network Address Translation (NAT): If you are using Reflection X in a network environment configured for NAT (sometimes referred to as IP masquerading, transparent proxying, or IP address overloading), read Technical Note 1513 before proceeding.
  • Virtual Private Networks (VPN): If you are connecting to the network through a VPN tunnel, read Technical Note 1580 before proceeding.
  • Some hosts include security features that prevent Reflection X (or other X servers) from establishing XDMCP connections. In some cases, this feature is enabled by default. Verify that your host is configured to permit XDMCP connectivity.

For details regarding this feature and Red Hat Linux 8.0, see technical note 1680.

Once you determine that your environment is compatible with XDMCP, proceed to C. Increase the Connection Timeout.

C. Increase the Connection Timeout

Increasing the connection timeout value may resolve connection problems, particularly when working in a low bandwidth environment. The connection timeout defines the number of seconds that Reflection X will try to make an XDMCP connection. By default, the value is set to 15 seconds.

Follow these steps to configure the connection timeout value.

  1. In the Reflection X Manager window, click File > New Client File. Then, in the New Connection window, click XDMCP Connection; click OK.
  2. Click Advanced to open the Advanced XDMCP Settings dialog box.
  3. In the Connection timeout field, enter the number of seconds that Reflection X should wait before timing out, and then click OK.
  4. Try to connect.

Did this work?

Yes. Save the .RXC file with the increased timeout setting for future use.

No. Proceed to D. Make a Direct XDMCP Connection.

D. Make a Direct XDMCP Connection

By default, Reflection X makes XDMCP connections using broadcast UDP packets. Some routers and firewalls are configured to block UDP or broadcast packets. Follow these steps to make a non-broadcast XDMCP connection:

  1. In the Reflection X Manager window, click File > New Client File. Then, in the New Connection window, click XDMCP Connection; click OK.
  2. In the Method drop-down list, select Direct.
  3. In the Host name field, enter your IP address or fully-qualified host name (including the domain name). For example, unixhost.mycompany.com.
  4. Click Connect.

Note: Depending on the configuration of your network, it may be possible to connect to your host using an unqualified host name; however, because this does not work in all network configurations, using a fully-qualified name is recommended.

Did this work?

Yes. Your router or firewall is most likely configured to block UDP broadcast packets. To workaround this problem, either use the Direct XDMCP connection; configure your workstation's firewall to permit UDP broadcasts, or contact your network administrator about configuring your router or firewall to permit UDP broadcasts.

No. Perform these steps:

    1. Follow the steps provided in the previous section, C. Increase the Connection Timeout, and increase the timeout setting for your Direct XDMCP connection. Try the connection again. If it still fails, proceed to step b.
    2. If connecting using the fully-qualified host name did not work, try using the host IP address. If the connection still fails, proceed to E. Ping the Host from the Workstation.

E. Ping the Host from the Workstation

Follow the steps below to determine if the host is available.

  1. Click Start > Run.
  2. In the Open field, enter command, and then click OK.
  3. At the command prompt, enter:
ping <host IP address or fully-qualified host name>

For example:

ping 111.222.33.44 or ping lindaj.mycompany.com

Did the host respond?

Yes. Proceed to F. Verify that Reflection X is Detecting the Correct IP Address.

No. Contact your system administrator to verify that the host is running and that you have the correct name or IP address for the host.

F. Verify that Reflection X is Detecting the Correct IP Address

Reflection X sends the workstation's IP address to the XDM daemon on the host. To verify that the workstation IP address is correct, follow the steps below.

  1. Click Start > Run.
  2. In the Open field, enter command, and then click OK.
  3. At the command prompt, enter the following command and note the IP address displayed:
ipconfig
  1. Note the IP address of the interface you want to use.
  2. In Reflection X Manager 14.1 SP1 or higher:
    1. Click Settings > View Settings.
    2. In the Search box, type “Detected IP”.
    3. Click the appropriate option for IPv4 or IPv6.
    4. Compare the IP address referenced under “Settings details” with the one you identified above.

In Reflection X Manager 14.0 – 14.1:

    1. Click Settings > Network.
    2. Compare the IP address referenced under "Autodetect network interface" with the one you identified above.

Do the IP addresses match?

Yes. Proceed to G. Ping the Workstation from the Host.

No. If the IP address automatically detected by Reflection X is not correct, manually enter the correct address under Settings > Network and return to D. Make a Direct XDMCP Connection.

G. Ping the Workstation from the Host

Verify that the host can access the workstation and correctly resolve the workstation's fully-qualified name. Incorrect resolution of the workstation name to the workstation IP address may cause XDMCP connections to fail.

An example of a fully-qualified workstation name is "lindaj.mycompany.com" (not "Lindaj"). Depending on the configuration of your network, it may be possible to connect to your host using an unqualified host name; however, because this does not work in all network configurations, using a fully-qualified name is recommended.

To locate your workstation's fully-qualified name, follow the appropriate steps for your operating system:

Windows 7: Click Start, right-click My Computer, and click Properties. The fully-qualified computer name will be under “Computer name, domain, and workgroup settings”.

Windows XP: Click Start, right-click My Computer, click Properties, and then select the Computer Name tab. The "Full computer name" field shows the fully-qualified workstation name.

Follow the steps below to ping your workstation from the host.

  1. Make a Telnet (or SSH if required) connection to your host using Reflection for UNIX and OpenVMS, or Reflection Workspace if you are running Reflection X as part of Reflection X 2011.

For Reflection for UNIX and OpenVMS:

    1. Click Start > Programs > Attachmate Reflection > Host - UNIX and OpenVMS.
    2. Click Connection > Connection Setup.
    3. Select the protocol to use, such as TELNET or SECURE SHELL.
    4. Enter Host name or IP address, and click Connect.
    5. Enter your credentials to log into the host.

For Reflection Workspace:

    1. Click Start > Programs > Attachmate Reflection > Reflection Workspace.
    2. Select VT Terminals, and click Create.
    3. Under Connection, select the protocol to use, such as Telnet or Secure Shell, enter Host name / IP address and click OK.
    4. Enter your credentials to log into the host.
  1. At the command prompt, enter the appropriate command for your host type to ping your workstation:

Linux: ping –c 3 <fully-qualified workstation name>

Solaris: ping –a <fully-qualified workstation name>

HP-UX: ping <fully-qualified workstation name> -n 3

Note: You may need to navigate to the /etc directory before issuing the ping command. To do so, enter cd /etc. If you are unable to navigate to /etc, or are still unable to use the ping command from that location, contact your system administrator for assistance.

A successful ping looks similar to this:

# ping lindaj.mycompany.com -n 3
PING LINDAJ.mycompany.com: 64 byte packets
64 bytes from 111.222.77.44: icmp_seq=0. time=1. ms
64 bytes from 111.222.77.44: icmp_seq=1. time=1. ms
64 bytes from 111.222.77.44: icmp_seq=2. time=1. ms

Did this work?

Yes. If you can ping the workstation by name, and the IP address returned is correct, proceed to H. Confirm that the Host XDM Daemon is Running.

If you can ping, but the IP address returned by ping is incorrect, you may have identified a DNS problem and should contact your network administrator for assistance. This problem must be resolved before proceeding.

Note: A temporary workaround for a DNS error is for the system administrator to add the correct workstation IP address and fully-qualified workstation name to the host's /etc/hosts file. Once this has been done, pinging from the host to the workstation should resolve properly. This method should only be used as a temporary workaround, especially in a DHCP network environment, because the /etc/hosts file would need to be updated each time the workstation obtains a new DHCP lease.

No. If you are unable to successfully ping using the workstation's name, try using the workstation's IP address in place of <fully-qualified workstation name>. If this works, proceed to the next section, "Reconfigure the Windows DNS suffix setting."

If a ping to the workstation IP address fails, contact your network administrator for assistance.

Note: Personal firewalls (running on the workstation) can be configured to disallow incoming echo requests (such as ping requests). If you have a personal firewall on your workstation, ensure that it is configured to allow incoming echo requests while you are testing with the ping command.

Reconfigure the Windows DNS suffix setting:

    1. Open the Local Area Connection Properties dialog box:

- Windows 7: Control Panel > Networking and Sharing Center > Local Area Connection > Properties

- Windows XP: Control Panel > Network and Internet Connections > Network Connections > right-click Local Area Connections > Properties

    1. Select Internet Protocol (TCP/IP). If multiple protocol versions are present, select the appropriate protocol version (IPv4 or IPv6).
    2. Click Advanced.
    3. On the DNS tab, check "Use the connection's DNS suffix in DNS registration," click OK, and exit Local Area Connection Properties.
    4. Re-try your host connection.

If changing this setting does not resolve the issue, clear the "Use this connection's DNS suffix in DNS registration" check box, and save the setting again. Proceed to H. Confirm that the Host XDM Daemon is Running.

H. Confirm that the Host XDM Daemon is Running

Follow these steps to confirm that the XDM daemon is running on your host.

  1. Make a connection to your host using either Reflection for UNIX and OpenVMS or Reflection Workspace, as in step G.
  2. At the command prompt, enter the appropriate command for your host system:
    Host
    Command
    UNIX host running CDE
    ps -eaf | grep dtlogin
    UNIX – Generic
    ps -eaf | grep login
    Linux host running KDM
    ps -eaf | grep kdm
    Linux host running GDM
    ps -eaf | grep gdm
    Linux host running XDM
    ps -eaf | grep xdm

A typical UNIX host response looks similar to the following; the path will vary depending on the host environment.

root 1355 1340 0 11:16 ? 14:25 /usr/dt/bin/dtlogin
guest 14519 14499 0 11:16 pts/10 00:00:00 grep dtlogin

In this example, the 'root' line confirms that the XDM daemon, entitled dtlogin, is running on the host, and is owned by the user 'root.'

A typical Linux host response looks similar to the following:

root 15099 1 0 Mar31 ? 00:00:09 /usr/bin/kdm -nodaemon
guest 15108 25084 09:07 00:00:00 grep kdm

In this example, the 'root' line confirms that the XDM daemon, entitled kdm, is running on the host, and is owned by the user 'root.' (The '-nodaemon' parameter does not indicate that the daemon is unavailable.)

Note the following:

  • The 'guest' line should be disregarded; it shows the status of the 'grep' command, not the status of the XDM daemon.
  • If you do not get a response after entering the appropriate command for your host, the XDM daemon is not currently running. Is the Daemon running?

Yes. Proceed to I. Check Port 177.

No. Contact your system administrator about the daemon. For more information on the XDM daemon, refer to the host Man pages.

Note: Whether the XDM Daemon is running or not, you can follow the steps below to manually start your X sessions. This workaround may allow you to use Reflection X while you are troubleshooting the XDMCP connection problem.

To manually start an X session without using the XDM Daemon

You can use the following steps to start an X desktop session if the XDM Daemon is not running.

  1. Start Reflection X (it can be minimized).
  2. Connect to your host using Reflection for UNIX and OpenVMS or Reflection Workspace.
  3. At the host command prompt, enter:

For UNIX:

/usr/dt/bin/Xsession -display <workstation IP or name>:0 &

For Linux:

/etc/X11/xdm/Xsession -display <workstation IP or name>:0 &

gnome-session & (no path needed)

startkde & (no path needed)

To use either of these last two Linux commands, the display variable has to be set first as neither supports the –display switch. (For example: "export SET DISPLAY=<workstation IP or name>:0")

Note: When you manually start an X session, some windows may stay open when you exit. To close these windows, reset your Reflection X server from the Action menu in the Reflection X Manager.

I. Check Port 177

To make an XDMCP connection, the network must be configured to allow the host and X server to communicate using UDP packets over port 177.

To verify that port 177 is open, contact your network administrator for assistance or use the Microsoft Portqry.exe command-line utility to determine if port 177 is available. For information about this utility, see Microsoft article 310099 at http://support.microsoft.com/default.aspx?scid=kb;en-us;310099.

Is Port 177 open on the host?

Yes. Proceed to J. Check Port 6000.

No. Work with your network administrator to identify why port 177 is unavailable. The port can be blocked by a network router and/or firewall. Connections using XDMCP are not possible until UDP port 177 is open.

J. Check Port 6000

X protocol typically uses port 6000 for communication between the host and Reflection X; therefore, if port 6000 is blocked between the host and the workstation, you will probably be unable to connect using Reflection X (one exception is when using Secure Shell). Follow the steps below to determine if port 6000 is available for the X protocol.

  1. Start Reflection X and minimize the Reflection X Manager window. When Reflection X is started, it attempts to open port 6000 on the workstation.
  2. Make a connection to the host using Reflection for UNIX and OpenVMS or Reflection Workspace.
  3. At the command prompt, type the following:
telnet <fully-qualified workstation name or IP address> 6000

For example: telnet lindaj.mycompany.com 6000

Port 6000 is open between the host and the workstation if the response looks similar to the following:

"Trying lindaj.mycompany.com...Connected to lindaj.mycompany.
com. Escape character is '^]'."

To exit the telnet connection, enter Ctrl+], and then enter quit.

Port 6000 is not open between the host and the workstation if the response is similar to one of the following strings:

"Timeout"
"Telnet: Unable to connect to remote host: Connection refused"
"Telnet: connect: Connection refused"
"Connection timed out"
"Could not open a connection to host on port 6000: Connection failed"

Is port 6000 open?

Yes. For further troubleshooting assistance, contact Attachmate Technical Support: http://support.attachmate.com/contact/.

No. If this is the case, check the following.

Firewalls:

    • Ensure that there is not a personal firewall blocking port 6000. See K. Check the Windows Firewall.
    • Some versions of the Cisco VPN Client contain a built-in firewall called Stateful Firewall. If this is running on your workstation, configure it to allow traffic on port 6000. For further information regarding this product, contact Cisco support or your company’s IT department.
    • Some versions of the Checkpoint VPN client contain a built-in firewall. If this is running on your workstation, configure the client to allow traffic over port 6000. For further information regarding this product, contact Checkpoint support or your company's IT department.
    • Talk to your network administrator to determine if a router or network firewall is blocking port 6000.

Note: It is not always obvious that a firewall is running, because it might be running in the background as a service. You may be able to determine if a firewall is running in this manner by looking in the Task Manager under the Application and Processes tabs, or by looking in the Services list. (Click Start > Control Panel > Administrative Tools > Services.)

Other Applications:

Determine if port 6000 is currently in use by an application other than Reflection X by entering netstat -a at the Windows Command Prompt (Programs > Accessories > Command Prompt).

    • If port 6000 is being used, you will see a line similar to the following:
TCP LINDAJ:6000 LINDAJ.mycompanye.com:0 LISTENING
    • If port 6000 is being used, and Reflection X is not running, another application is using port 6000 (and possibly disallowing Reflection X access to the port). Determine what application is using the port and disable it.
    • Depending on which operating system you are using, you may be able to use a freeware utility, such as Active Ports by SmartLine Inc, to determine what application is using this port. This utility can be obtained by doing a web search for "active ports smartline".

If you are successful in opening port 6000, try to connect using XDMCP again. If still unable to connect, proceed to Contact Attachmate.

K. Check the Windows Firewall

Follow the steps below to view and modify the Windows Firewall settings

  1. Click Start > Control Panel > Windows Firewall.

Is the firewall enabled?

Yes. Before you can make an XDMCP connection, you must include Reflection X in the list of allowed programs. (You may also choose to disable the firewall temporarily for testing, or permanently if this is appropriate in your environment.)

Add Reflection X to the list of allowed programs:

On Windows 7:

    1. Click Start > Control Panel > Windows Firewall.
    2. From the left pane, click Allow a program or feature through Windows Firewall.
    3. Click Allow another program.
    4. Browse to select the Reflection X program file. (The default location is either C:\Program Files\Attachmate\Rx.exe, or C:\Program Files\Attachmate\R14\Rx.exe.)

On Windows XP:

    1. Click Start > Control Panel > Windows Firewall.
    2. Click Exceptions.
    3. Click Add Program.
    4. Browse to select the Reflection X program file. (The default location is either C:\Program Files\Attachmate\Rx.exe, or C:\Program Files\Attachmate\R14\Rx.exe.)

No. Return to J. Check Port 6000 and proceed through the firewalls list. Or, if you have already done this, proceed to Contact Attachmate.

L. Record and Review a Network Trace

If you are familiar with network traces, take a network trace of your XDMCP connection attempt and refer to the following Reflection XDMCP connection details while reviewing the trace.

Note: If you do not have a network trace utility, you may want to try one of the free online utilities, such as Wireshark, a multi-platform network protocol analyzer available at http://www.wireshark.org/.

In the network trace, you should see something similar to the following, showing Reflection X and the host negotiating and maintaining the connection. (Exactly how the information is presented depends on the trace utility you are using.)

The first set of entries show Reflection X connecting to port 177 on the host using XDMCP protocol over UDP. Communication is established between a random port number on the PC and port 177 on the host. During this process Reflection X (on the PC) provides the host with information about which port number it is configured to use for X11 protocol. Typically, this is port 6000.

Note: If there is a firewall between the PC and host, the firewall must be configured to allow UDP packets from the PC to the host on port 177.

Source
Destination
Protocol
Packets
Information
PC
Host
XDMCP
UDP
Query
Host
PC
XDMCP
UDP
Willing
PC
Host
XDMCP
UDP
Request
Host
PC
XDMCP
UDP
Accept
PC
Host
XDMCP
UDP
Manage

The second set of entries shows the host sending a SYN request to the PC on the port that Reflection X specified for X11 protocol (typically port 6000), using TCP. The following SYN and ACK communications are made between a random host port and the port Reflection X specified for X11 protocol.

Note: If there is a firewall between the PC and host, the firewall must be configured to allow TCP packets from the host to the PC on port 6000 (or whichever port Reflection X is listening on for X11 traffic).

Source
Destination
Protocol
Packets
Information
Example
Host
PC
X11
TCP
specified host port number > X11 port number [SYN]
38197 > 6000 [SYN]
or
38197 > X11 [SYN]
Note: If the default X11 port is being used, 6000, "X11" may be displayed instead of 6000.

PC
Host
X11
TCP
X11 port number > specified host port number [SYN, ACK]
X11 > 38197 [SYN, ACK]
Host
PC
X11
TCP
specified host port number > X11 port number [ACK]
38197 > X11 [ACK]

Finally, the host and Reflection X begin communicating using X11 protocol over TCP on the ports established during the SYN-ACK process above. No additional ports need be opened in the firewall.

Source
Destination
Protocol
Packets
Information
Host
PC
X11
TCP
Initial connection request
PC
Host
X11
TCP
Initial connection reply
Host
PC
X11
TCP
Requests
PC
Host
X11
TCP
Responses
PC
Host
X11
TCP
Events
PC
Host
X11
TCP
Errors

Contact Attachmate

If you have worked through all of the troubleshooting suggestions in this document and are still having difficulties, contact Attachmate Technical Support, http://support.attachmate.com/contact/, for further assistance.

Related Technical Notes
1513 Reflection X 14.x and Network Address Translation (NAT)
1530 Startup Parameters for Reflection X 14.x
1580 Connecting through a VPN (Virtual Private Network) with Reflection X 14.x
1680 XDMCP Connections to Red Hat Linux 7.0 or Higher
9992 Reflection X Technical Notes

Did this technical note answer your question?

           



Need further help? For technical support, please contact Support.