Remote X11 understanding

Suppose we have a local machine (windows/linux), wanna do some deep learning training or data analysis in a remote linux server to . To show images like plt.plot() & plt.show() in local machine we need X11 forwarding which directly renderes images in local machine.

Ok, first step we should connect to a remote linux server from our local machine. Supposing using SSH connection in MobaXterm, we need a private key in local machine and a public key in remote server. Then ssh user@ip -X. The -X here enables X11 forwarding. W.o. -X, any plot command like plt.plot() & plt.show() wouldn’t be shown in local machine.
Next, what we need to care about is the $DISPLAY variable in remote server. Using echo $DISPLAY in local terminal to check the value. If the -X args is enabled, the it will output something like localhost:10.0, wehre localhost refers to hostname of your computer (I’m not sure whether it refers to remote server or local machine). 10 display name, 0 screen name. Generally speaking, a display is composed of a screen, a keyboard and a mouse.
Now use xclock in your terminal to test GUI.

configure on vscode

Install remoet X11 and remote X11 (ssh)
Use echo $DISPLAY in vscode terminal to check display value
If display value is none, add export DISPLAY=localhost:10.0 to ~/.bashrc and source ~/.bashrc.
Open vscode configuration ctrl+shit+p and find out remote X11 settings, set up the display 10, screen 0.
Using xclock to test

Notice

The default X11 forwarding can’t render images or videos generated by python gym library. Because, the X11 forwarding only supports OPENGL1.5 and gym requires higher version. To render gym images, it involves a technique named “VirtualGL”. It is complicated, I need some time to figure it out. Maybe next time.

Configure GUI backend for singularity

Sometimes although the RemoteX11 is setup for ssh login, the matplotlib in singularity cannot show as expected. A common warning is UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure..

Solution for remote node:
1. For a remote node, we have to ssh with x11 forwarding, -X.
2. For a interactive node or computation node on remote cluster, we have to apply with arg --x11.
3. To show matplotlib.pyplot -> plt.switch_backend('tkAgg').
The solution only works for X11 forwarding. Some applications require openGL library. Now let’s try to sort it out.

Trying to understand the openGL

remote server with openGL is still not solved:frowning:

众所周知，OpenGL作为图形界的工业标准，其仅仅定义了一组2D和3D图形接口API，而对于窗口管理、IO消息响应等并没有规定。也就是说，OpenGL依赖各平台提供用于渲染的context以及具体实现方式，而各平台提供的实现不尽相同。这些实现主要有：Windows平台下的WGL、Linux下的Mesa/GLX、Mac OS X下的Cocoa/NSGL，以及跨平台的GLUT、GLFW、SDL等等。

Mesa是Linux下的OpenGL实现。它提供了对AMD Radeon系列、Nvidia GPU、Intel i965, i945, i915以及VMWare虚拟GPU等多种硬件驱动的支持，同时也提供了对softpipe等多种软件驱动的支持。Mesa项目由Brian Paul于1993年8月创建，于1995年2月发布了第一个发行版，此后便受到越来越多的关注，如今Mesa已经是任何一个Linux版本首选的OpenGL实现。

GLX则是在Linux上用于提供GL与窗口交互、窗口管理等等的一组API。它的作用与Windows的WGL、Mac OS X的AGL以及针对OpenGL ES的EGL相似。在Linux上，窗口创建、管理等API遵循X Window接口，而GLX提供了OpenGL与X Window交互的办法。因此GLX也可以运用于其他使用X Window的平台，例如FreeBSD等。

However, OpenGL by itself is not an API, but merely a specification. It is just a description of what exactly the result/output of each function should be and how it should perform. It is already implemented inside your driver, by manufacturers, following the specification. Hence, there is no such thing as “installing” OpenGL. But, we do need to install libraries, that would help us interact with the operating system to access the implementation and set up the windowing system & OpenGL context.

GLFW

GLFW is an Open Source, multi-platform library for OpenGL, OpenGL ES and Vulkan development on the desktop

OpenGL Setup: GLFW Library

Before you start with creating stunning graphics, you need to initialize an OpenGL context and create an application window to draw in. We’ll do this using a popular C library: GLFW(Graphics Library Framework). This library also helps us handle the input from the joystick, keyboard, and mouse.

GLEW

The OpenGL Extension Wrangler Library (GLEW) is a cross-platform open-source C/C++ extension loading library. GLEW provides efficient run-time mechanisms for determining which OpenGL extensions are supported on the target platform.

xserver

Xserver

In this example you have a local X11-server with two “screens” on your hostA. Usually there would be only one server with one screen (:0.0), which spans across all your monitors (makes multi-monitor applications way easier). hostB has two X servers, where the second one has no physical display (e.g. virtual framebuffer for VNC). hostC is a headless server without any monitors.

terminal 1a, 2a, 5a, 6a: If you open a local terminal, and set the display to :0.0 (default) or :0.1, the drawing calls for your graphical programs will be sent to the local X server directly via the memory.

terminal 1b, 5b: If you ssh onto some server, usually the display will be set automatically to the local X server, if there is one available. Otherwise, it will not be set at all (reason see terminal 3).

terminal 2b, 6b: If you ssh onto a server, and enable X11-forwarding via the “-X” parameter, a tunnel is automatically created through the ssh-connection. In this case, TCP Port 6010 (6000+display#) on hostB is forwarding the traffic to Port 6000 (X server #0) on hostA. Usually the first 10 displays are reserved for “real” servers, therefore ssh remaps display #10 (next user connecting with ssh -X while you’re logged in, would then get #11). There is no additional X server started, and permissions for X-server #0 on hostA are handled automatically by ssh.

terminal 4: If you add a hostname (e.g. localhost) in front of the display/screen#, X11 will also communicate via TCP instead of the memory.

terminal 3: You can also directly send X11 commands over the network, without setting up a ssh-tunnel first. The main problem here is, that your network/firewall/etc. needs to be configured to allow this (beware X11 is practically not encrypted), and permissions for the X server need to be granted manually (xhosts or Xauthority).

Xorg

Xorg (commonly referred to as simply X) is the most popular display server among Linux users. Its ubiquity has led to making it an ever-present requisite for GUI applications, resulting in massive adoption from most distributions

mesa

Solutions for openGL in remote server

First and the most important is the remote server should support openGL. For our HPC server, openGL is not supported, so there is no way rendering any 3D plots in remote server in a terminal way.
How to work around it? For graham, it is better to login grad-vdi.computecanada.ca. For other clusters, using Paraview is a good idea.

For grad-vdi

1
2
3
4


local:
    1. vncviewer
    2. grad-vdi.computecanada.ca
    3. login and module load stdenv

For Paraview

1
2
3
4
5
6
7
8


Remote:
    1. module load gcc/9.3.0 paraview-offscreen-gpu/5.10.0
    2. unser DISPLAY
    3. pvserver
local:
    1. ssh username@dns -L 11111:gra947:11111
    2. paraview
    3. `File->connect->add server` with port `11111`

Remote X11 understanding#

configure on vscode#

Notice#

Configure GUI backend for singularity#

Trying to understand the openGL#

Solutions for openGL in remote server#

For grad-vdi#

For Paraview#