Bugs:realvnc4/pixel format is unpredictable after refreshing and suddenly changing pixel format

This bug apply to realvnc 4.0 to 4.1.1. The bug may demonstrate the flaw in VNC protocol, which don't have any mechanism to ensure the synchronization of the pixel format between client and server.

Let's see how to reproduce this bug first, and then see how pixel format is out of sync.

How to reproduce

Run the following command in some terminal, "terminal1", in this example,

terminal1 $ Xvnc :1 -nolisten tcp -auth /dev/null -localhost -SecurityTypes=None -geometry 800x600 -depth 24 &
terminal1 $ DISPLAY=:1 xterm -e perl -e '$|=1; for(;;){print"."; select(undef,undef,undef,0.001); }' &

Then open another terminal, "terminal2", run vncviewer,

terminal2 $ vncviewer :1 -shared -AutoSelect=0 -ViewOnly=1 -FullColor

VNC Viewer Free Edition 4.1.1 for X - built Jan  6 2007 00:46:26
Copyright (C) 2002-2005 RealVNC Ltd.
See http://www.realvnc.com for information on VNC.

Fri Dec  7 20:11:20 2007
 CConn:       connected to host localhost port 5901

Fri Dec  7 20:11:21 2007
 CConnection: Server supports RFB protocol version 3.8
 CConnection: Using RFB protocol version 3.8
 TXImage:     Using default colormap and visual, TrueColor, depth 24.
 CConn:       Using pixel format depth 24 (32bpp) little-endian rgb888
 CConn:       Using ZRLE encoding

Switch back to "terminal1", suspend the Xvnc,

terminal1 $ killall -TSTP Xvnc

Then switch to vncviewer x11 window, refresh screen, and change colour level,

1. vncviewer x11: press F8 --> select "Refresh screen"
2. vncviewer x11: press F8 --> select "Options..." --> select "Very low (8 colours)" in "Encoding and Colour Level:" --> click "Ok"

Switch to "terminal1", resume Xvnc,

terminal1 $ killall -CONT Xvnc

After Xvnc is resumed, vncviewer then fail like this,

terminal2 $ vncviewer :1 -shared -AutoSelect=0 -ViewOnly=1 -FullColor
<--cut-->
...
<--cut-->
 CConn:       Using ZRLE encoding

Fri Dec  7 20:11:45 2007
 CConn:       Using pixel format depth 3 (8bpp) rgb111
vncviewer: ../rfb/zrleDecode.h:196: void rfb::zrleDecode8(const rfb::Rect&, rdr::InStream*, rdr::ZlibInStream*, rdr::U8*, rfb::CMsgHandler*): Assertion `len <= end - ptr' failed.
Aborted

Notes

• result on other version,
  • vncviewer 3.3.7 --> Xvnc 3.3.7 : can't reproduce
  • vncviewer 4.1.1 --> Xvnc 3.3.7 : can't reproduce
  • vncviewer 3.3.7 --> Xvnc 4.0 : reproducible
  • vncviewer 4.1.1 --> Xvnc 4.0 : reproducible

Cause

This is because the client and server fail to synchronize the pixel format, which is depicted as follow,

                                       server send FramebufferUpdate0
                                       server is suspended
client receive FramebufferUpdate0
client send FramebufferUpdateRequest1:incremental=true

user request refresh screen
client send FramebufferUpdateRequest2:incremental=false
user change colour level

                                       server is resumed
                                       server receive FramebufferUpdateRequest1:incremental=true
                                       server send FramebufferUpdate1
client receive FramebufferUpdate1
client change its internal pixel format to new pixel format
client send SetPixelFormat
client send FramebufferUpdateRequest3:incremental=false
           (non-incremental since pixel format changed)
client is now expecting new pixel format from server
                                       server receive FramebufferUpdateRequest2:incremental=false
                                       server send FramebufferUpdate2 with old pixel format
client receive FramebufferUpdate2 with old pixel format
client interprete FramebufferUpdate2 as new pixel format
client mis-interprete pixel format, and then fail