The monitor server command relies on a thread dump utility, which provides a thread-safe method of extracting information from ENOVIA Live Collaboration threads without risking the stability of the system. At various milestones during their execution, ENOVIA Live Collaboration threads record their status, which can then safely be collected and "dumped". For each thread that is actively doing work, the stack of ENOVIA Studio Customization Toolkit calls, program invocations, trigger invocations, running MQL commands, as well as the length of time since the thread entered each level of the stack, can be reported. This gives a complete picture of what each thread is doing "right now," which user is doing the work, and how long the thread has been working on the current request. You can query a sluggish server to see what requests are being processed and which of them might account for the problem. On-demand thread details can be output in three ways:
Each output method is described in the sections that follow. The last section shows sample output. Monitor Server commandYou can issue the monitor server command against a running server through the emxRunMQL admin tool, or by connecting directly to the server from a separate, stand-alone Java program or custom JSP. See "monitor Command" in the Reference Commands chapter of the MQL Guide. UNIX Kill -QUIT commandThe UNIX kill -QUIT command has always provided a dump of the Java threads currently executing when issued against a running Java process, such as an application server or RMI server. This is a function of the JVM. It does not stop the server; it merely reports the program stack for the current Java threads on stdout. These Java stacks do not by default extend down into native libraries such as the ENOVIA Live Collaboration kernel library. However, when kill -QUIT is issued on an ENOVIA Live Collaboration Server on Unix, the specific thread details (both C++ and Java) are prepended to the output. This is supported on all ENOVIA Live Collaboration Server Java 1.4 deployments on Unix. Solaris and HP also support this interface with Java 1.3. Note: You must include the process ID (PID) as follows: kill -QUIT PID The PID is either the JVM's RMI daemon (the child process of RMI) or the application server's JVM PID if running in RIP. For RMI Gateway configurations, run the command against all of the RMI's JVM PIDs. The output goes to stdout, so you should start the server with the nohup option or redirect the output to a file. Note: While the kill command is not available on Windows, you can use Ctrl+Break from a Windows application server's console running in RIP mode to get a thread dump. TCP ListenerIf your servers do not support kill -QUIT, or if you want to set up a program to ping multiple servers periodically, you must ensure that your servers are listening for server monitor requests. It may also be useful if Java threads (RMI daemon or application servers) seem not to be responding. In order to set up a listener thread, ENOVIA Live Collaboration needs a distinct TCP port specification for each ENOVIA Live Collaboration Server that is running on the same machine. By default, output is sent to a server-assigned port, and the port number is identified in mxtrace.log: Notice #4000047 Listening on MX_DEBUG_PORT 4465 You can connect to this port from a separate process (with telnet, for instance) and get the monitor server output. You can also provide a port number by setting MX_DEBUG_PORT in the environment from which the server is started, in rmireg.sh (UNIX) or rmireg.bat (Windows). By default it is set to 0, so that the system assigns an available port. If you set it to -1, listening on a port is disabled. For RMI gateway configurations, it is important to specify a different port number for each of the RMI servers. For example, if you have a gateway set up on port 1099 with RMI servers on port 1100 and 1101, the entries shown below in bold (using a port number of your choice) should be added to rmireg.sh or rmireg.bat: MX_DEBUG_PORT=4465
export MX_DEBUG_PORT
${JAVA_PATH}/rmid $1 $JAVA_OPTIONS $JAVA_LIB -C-Djava.rmi.server.codebase=file:$CODEBASE -C-Djava.rmi.server.useCodebaseOnly=true -J$JAVA_SECURITY -port 1100 -log /RMIINSTALL/logs/RMI1100 & Similarly for the RMI process on port 1101: MX_DEBUG_PORT=4466
export MX_DEBUG_PORT
${JAVA_PATH}/rmid $1 $JAVA_OPTIONS $JAVA_LIB -C-Djava.rmi.server.codebase=file:$CODEBASE -C-Djava.rmi.server.useCodebaseOnly=true -J$JAVA_SECURITY -port 1101 -log /RMIINSTALL/logs/RMI1101 & In addition, you should configure a separate stdout file for each server in the gateway. For example, add the bolded part at the end of the lines that set the Java options, as follows: ${JAVA_PATH}/rmid $1 $JAVA_OPTIONS $JAVA_LIB -C-Djava.rmi.server.codebase=file:$CODEBASE -C-Djava.rmi.server.useCodebaseOnly=true -J$JAVA_SECURITY -port 1100 -log /RMIINSTALL/logs/RMI1100 2>&1 >> rmi_stdout_1100.log & Similarly for the RMI process on port 1101: ${JAVA_PATH}/rmid $1 $JAVA_OPTIONS $JAVA_LIB -C-Djava.rmi.server.codebase=file:$CODEBASE -C-Djava.rmi.server.useCodebaseOnly=true -J$JAVA_SECURITY -port 1101 -log /RMIINSTALL/logs/RMI1101 2>&1 >> rmi_stdout_1101.log & For RMI gateway on Windows running as a service, it is recommended that you use the default setting for MX_DEBUG_PORT (0) to ensure a port is assigned to all RMI servers. The MX_DEBUG_PORT is run on a thread separate from ENOVIA Live Collaboration. Consequently, when mxAudit.log is generated at system startup, there is the possibility that the MX_DEBUG_PORT listener thread is not yet done setting up the port. In this case, mxAudit.log will contain a message such as the following: MX_DEBUG_PORT listener not yet running at time of audit - check mxtrace.log. In any case, if an ENOVIA Live Collaboration Server is unable to start a listener thread, it reports the failure in the mxtrace.log file, together with the port number that failed. If it succeeds, it also puts a confirmation notice into mxtrace.log, including the port number that it is listening on. In a gateway environment there is a "listening on port" entry in mxtrace.log for each RMI gateway server started. Example mxtrace.log entries follow: If listen fails: #0 System Error #4000046 Failed to listen on MX_DEBUG_PORT 1099 (port in use) Other informational messages: Notice #4000047 Listening on MX_DEBUG_PORT 4465 Notice #4000049 MX_DEBUG_PORT explicitly disabled. OutputThe MQL monitor server command returns output to the caller as a string, or dumped to a plain text file. The signal interface will dump thread information to stdout. The format of this information may change, possibly significantly, in future releases as further phases of the server monitor are developed. The output represents the state of all threads executing in the ENOVIA Live Collaboration library, (or a subset if the age option is specified) at the time the request is made. The following types of information are reported in the phase 3 core server monitor. For each thread reported on, a stack consisting of these types of entries is output.
All entries also include the total processing time and any resulting mxtrace.log entries. Every thread running in the kernel is the result of an ENOVIA Studio Customization Toolkit call that originates from an ENOVIA product JSP or servlet, a JPO, a custom jsp, Web Navigator, or a custom application. Therefore, for each thread reported by the monitor, the first entry is always: <1- ADK Trace Entry>. The numbers following the opening bracket "<" indicate the sequence in which the entries occurred. Unlike programming stacks, the entries are listed oldest to youngest. For instance you might see the following headers in the output: <1- ADK Verbose Trace Entry>... <2- Launching Trigger Program>... <3- Launching Program Object>... <4- Launching Java Object>... <5- ADK Verbose Trace Entry>... Notice that entries 1 and 5 are both ENOVIA Studio Customization Toolkit entries, because the JPO launched in entries 3 and 4 is making an ENOVIA Studio Customization Toolkit call back into ENOVIA Live Collaboration. Sample OutputNote: Be aware that the format of the below information will change, possibly significantly, in future releases. *************************************************** eMatrix Server Trace - Version 10.6.2.0 - 11/21/2005 18:18:13 GMT: >>>> >>>> Tells how long the server has been up. >>>> eMatrix Active: 1 hour 5 minutes >>>> >>>> Thread #1 entered the server as user creator to evaluate a and has >>>> been doing so for 0 seconds. Query fields are provided >>>> Thread #1: t@053, ref=0x485b98 session=mx113259838319617870931 ms=78e2fe00x08 <Start Stack Trace> <1 - ADK Verbose Trace Entry> stateless dispatch for evaluateSelect.bosQuery executing Active: 0 seconds parameters: bosContext _cntx: user: creator depth: 1 session id: mx113259838319617870931 bosString _name: bosQuerySt _query: bosQueryClauseSt clause bosString typePattern CaseOutline bosString namePattern * bosString revisionPattern * bosString latticePattern * bosString ownerPattern * bosString where name matchlist "*mon-yen,*tue-yng" "," && ((current ~= "Active") && ("attribute[RAPTestName]" ~~ "**") && ("attribute[Last Name]" ~~ "**") && ("relationship[Employee].from.name" ~= "Acme Technologies")) bosString fullTextSearch bosString fullTextFormat short limit 100 uint8 expandTypes 1 uint8 queryTrigger 0 bosString description bosDataList icon 0 entries bosStringList _objectSelect: 5 entries bosString str attribute[First Name] bosString str attribute[Last Name] bosString str name bosString str id bosString str relationship[CaseOutlineToRAPPackageData].from stacktrace: at com.MatrixOne.jdl.bosQueryShim.evaluateSelect(bosQueryShim.java:102) at matrix.db.Query.select(Query.java:821) at ContextTest.doQuery(ContextTest.java:109) at ContextTest.run(ContextTest.java:64) >>>> >>>> If there are any warnings/errors that have occurred in this thread >>>> they will be shown here. >>>> Error Stack: Warning #1500568: Where clause is not valid; contains term 'attribute[Last Name]' referring to a non-existent attribute. <End Stack Trace> >>>> >>>> Thread #2 entered the server as user creator to execute an mql command >>>> and has been doing so for 0 seconds. >>>> The mql command is provided - 'exec prog JPOTest' >>>> Thread #2: t@024, ref=0x36e268 session=mx1132597948702605645 ms=1a3a380x08 <Start Stack Trace> <1 - ADK Verbose Trace Entry> stateless dispatch for executeCmd.bosMQLCommand executing Active: 0 seconds parameters: bosContext _cntx: user: creator depth: 3 session id: mx1132597948702605645 bosString _cmd: exec prog JPO_Test stacktrace: at com.MatrixOne.jdl.bosMQLCommandShim.executeCmd(bosMQLCommandShim.java: 32) at matrix.db.MQLCommand.executeCommand(MQLCommand.java:90) at DiagTestProg.run(DiagTestProg.java:55) Error Stack: <2 - Executing MQL Command> Active: 0 seconds cmd=exec prog JPO_Test Error Stack: >>>> >>>> entries 3,4 show the launching of the program >>>> <3 - Launching Program Object> Active: 0 seconds Name=JPO_Test, user=, invocation=program Global env at program launch: Local env at program launch: APPLICATION=BOS INVOCATION=program MATRIXHOME=/local/home/dvlp/ksmith/rmi10550 MATRIXPATH= PATH=/usr/j2se/bin:/local/home/dvlp/ksmith/rmi10550/bin/solaris4:/ usr/dt/bin:/usr/openwin/bin:/usr/openwin/bin/xview:/sbin:/bin:/usr/ bin:/usr/sbin:/opt/SUNWspro/prod/bin:/usr/ucb:/usr/ccs/bin:/etc:/usr/ etc:/usr/etc/install:.:/sqa/netlib:/usr/local/bin:/usr/local/bin/X11:/ app/qausr/matrix-install/scripts:/app/verity/k2/k2/_ssol26/bin Deferred: false Error Stack: <4 - Launching Java Object> Active: 0 seconds progName: JPO_Test constructor parameters: className: JPO_Test_mxJPOA9VrXAAAAAEAAAAG methodName: mxMain method parameters: Error Stack: >>>> >>>> The program calls back through the ADK interface to execute another MQL >>>> command - 'monitor server' >>>> <5 - ADK Verbose Trace Entry> stateless dispatch for executeCmd.bosMQLCommand executing Active: 0 seconds parameters: bosContext _cntx: user: creator depth: 3 session id: mx1132597948702605645 bosString _cmd: monitor server >>>> >>>> And finally, the java stack trace that led to the initial ADK call is also >>>> provided.(This is not provided by default - ADK Origin trace must be enabled) >>>> stacktrace: at com.MatrixOne.jdl.bosMQLCommandShim.executeCmd(bosMQLCommandShim.java: 32) at matrix.db.MQLCommand.executeCommand(MQLCommand.java:90) at JPO_Test_mxJPOA9VrXAAAAAEAAAAG.mxMain(JPO_Test_mxJPOA9VrXAAAAAEAAAAG.j ava:25) at java.lang.reflect.Method.invoke(Native Method) at matrix.db.MatrixClassLoader.invokeObject(MatrixClassLoader.java:383) at matrix.db.MatrixClassLoader.invokeObject(MatrixClassLoader.java:400) at com.MatrixOne.jni.MatrixKernel.statelessDispatch(Native Method) at com.MatrixOne.jdl.rmi.bosMQLCommandImpl.executeCmd(bosMQLCommandImpl.j ava:46) at java.lang.reflect.Method.invoke(Native Method) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:236) at sun.rmi.transport.Transport$1.run(Transport.java:147) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:143) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:46 0) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport. java:701) at java.lang.Thread.run(Thread.java:479) >>>> >>>> No errors for Thread #2 >>>> Error Stack: <End Stack Trace> >>>> >>>> Thread #3 is also executing an MQL command - 'promote bus PART-u Test 0' >>>> Thread #3: t@021, ref=0x36e2e0 session=mx113259964028925613505 ms=42f1b80x08 <Start Stack Trace> <1 - ADK Verbose Trace Entry> stateless dispatch for executeCmd.bosMQLCommand executing Active: 4.00 seconds parameters: bosContext _cntx: user: creator depth: 1 session id: mx113259964028925613505 bosString _cmd: promote bus PART-u Test 0; Error Stack: Warning #1500318: The ConnectString in your connection (bootstrap) file does not match the ConnectString of any server object defined in the database <2 - Executing MQL Command> Active: 4.00 seconds cmd=promote bus PART-u Test 0; Error Stack: >>>> >>>> The promotion launches a trigger. Trigger macros and RPE values are shown >>>> <3 - Launching Trigger Program> Active: 4.00 seconds TriggerState active=true owner Oid=212401520 name=JPO_DiagTest args= info=PromoteCheck Error Stack: <4 - Launching Program Object> Active: 4.00 seconds Name=JPO_DiagTest, user=, invocation=check Global env at program launch: Local env at program launch: ACCESSFLAG=True APPLICATION=BOS AUTOPROMOTE=False CHECKACCESSFLAG=True CURRENTSTATE=one DESCRIPTION= ENFORCEDLOCKINGFLAG=False EVENT=Promote HASACTUALDATE=True HASSCHEDULEDATE=False INVOCATION=check ISCURRENT=True ISDISABLED=False ISENABLED=True ISOVERRIDDEN=False ISREVISIONABLE=True ISVERSIONABLE=True LATTICE=unit1 LOCKER= LOCKFLAG=False MATRIXHOME=/local/home/dvlp/kkohn/rmi10550 MATRIXPATH= NAME=Test NEXTSTATE=final OBJECT="PART-u" "Test" "0" OBJECTID=32541.36365.3240.64880 OWNER=creator PATH=/usr/j2se/bin:/local/home/dvlp/kkohn/rmi10550/bin/solaris4:/ usr/dt/bin:/usr/openwin/bin:/usr/openwin/bin/xview:/sbin:/bin:/usr/ bin:/usr/sbin:/opt/SUNWspro/prod/bin:/usr/ucb:/usr/ccs/bin:/etc:/usr/ etc:/usr/etc/install:.:/sqa/netlib:/usr/local/bin:/usr/local/bin/X11:/ app/qausr/matrix-install/scripts:/app/verity/k2/k2/_ssol26/bin POLICY=simple-u REVISION=0 STATENAME=one TIMESTAMP=Mon Nov 21, 2005 2:01:30 PM EST TRIGGER_VAULT=unit1 TYPE=PART-u USER=creator VAULT=unit1 Deferred: false Error Stack: <5 - Launching Java Object> Active: 4.00 seconds progName: JPO_DiagTest constructor parameters: className: JPO_DiagTest_mxJPOAeDjNAAAAAEAAAAC methodName: mxMain method parameters: >>>> >>>> No errors for Thread #3 >>>> Error Stack: <End Stack Trace> Monitored 3 active eMatrix threads *************************************************** Since it passes large strings across the ENOVIA Studio Customization Toolkit connection for every server call, thereby affecting performance, ENOVIA Studio Customization Toolkit Origin trace output is not included in server monitor output by default. However, if diagnostic efforts would be significantly aided by knowing the appserver-tier Java stack (JSP's and beans) that initiated certain ENOVIA Studio Customization Toolkit calls, it can be useful, and is therefore included in server monitor output when enabled. With the introduction of Flash DB, ENOVIA Live Collaboration supports multi/hyper threaded CPUs. Hyper-threading support provides better utilization of multi-core CPUs which are becoming common place. This support introduces background threads that do work which, in prior releases, occurred in a single threaded fashion. The result is improved performance and higher throughput. The threads consume very little memory and are transparent to the JVM and/or application server. Server monitor output has been enhanced to include these threads, as in the example below: >>>>
>>>> Thread #1 is the parent thread of "Background Task" example below.
>>>>
Thread #1: t@155, ref=0x855400,
session=mx16717111951ee8e:mx1688703977961823, ms=0x01e28c00
<Start Stack Trace>
<1 - ADK Verbose Trace Entry>
stateless dispatch for executeCmd.bosMQLCommand executing
Active: 0 seconds
User:
creator
Session:
mx16717111951ee8e:mx1688703977961823
Parameters:
bosContext _cntx:
user:
creator
depth:
1
session id:
mx16717111951ee8e:mx1688703977961823
bosString _cmd:
list prog
Error Stack:
>>>>
>>>> ADK call MQLCommand issued "list prog"
>>>>
<2 - Executing MQL Command>
Active: 0 seconds
Command:
list prog
Error Stack:
<End Stack Trace>
Thread #2: t@158, ref=0x80f918,
session=mx16717111951ee8e:mx1688703977961823, ms=0x01e28c00
<Start Stack Trace>
<1 - Launching Background Task>
Active: 0 seconds
>>>>
>>>> Task type: "FlashNextTask" indicates cache is being read. Parent
thread id is found
>>>> in thread #1 above.
>>>>
Task type: FlashNextTask
Parent Thread: t@155
Priority: high
Reference count: 2
Error Stack:
<End Stack Trace>
Monitored 2 active threads
The defined background task types are:
|