Introduction

Joanju.Analyst sits on top of a software stack including Java, Proparse, Tomcat, MySQL, and ProRefactor.

Contents

Setup
Requirements
Tomcat
The .joanju Configuration Directory
Proparse
ProRefactor Configuration Files
Server Properties File
MySQL
Reviewing the Administrative Page
Build/Refresh
Configuration Scripts
Unresolved Calls Report

Mismatched Parameters Report
Name Propagation Across Expressions
Eclipse Plug-in Client
Call Graphing Limitations

Requirements

• Web browser: Firefox or Opera. IE does not work.
• Java: Version 5 or later.
• Tomcat: Recent stable release.
• MySQL: Recent stable release.
• Proparse: Latest available release. Must be downloaded separately (different binaries for each operating system).

Tomcat

Analyst is a Java web application, and must be installed into Apache Tomcat or similar. (Only Tomcat has been tested.) Unzip the Analyst distribution file into your Tomcat webapps directory. This creates an analyst directory. If Tomcat is running on port 8080 (depends on your configuration) then the URL for Analyst will be:

  http://127.0.0.1:8080/analyst/

You are free to change the directory name from analyst to anything you like, and the URL would change accordingly. (Useful for multiple installations of Analyst under one Tomcat server.)

The Java options for Tomcat startup must be made larger than their default values. Maximum physical memory (-Xmx) should be 128M at a minimum, or larger if you have sufficient physical memory on your computer. The stack size (-Xss) should be 2M or 3M. I've found 2M is enough for most applications.

On Windows, I am running Tomcat 6. It has a configuration window, with a 'Java' tab. The 'Maximum memory pool' field is for the physical memory value described above, and the 'Thread stack size' field is for the stack size value described above.

On Ubuntu Linux, I edited JAVA_OPTS in the /etc/defaults/tomcat55 file to adjust the Java memory settings:

  JAVA_OPTS="-Djava.awt.headless=true -Xss3M -Xmx600M"

The .joanju Configuration Directory

Into your web application directory (ex: TOMCAT/webapps/analyst) create a directory named .joanju. You will add various mandatory and optional configuration files to this directory. Analyst upgrades will never overwrite the contents of this directory.

Proparse

Copy the Proparse binary into the web application .joanju directory so that it looks like this: .joanju/proparse.dll, regardless of which operating system you are using. For example, if you are installing onto a Linux computer, you would copy your downloaded libproparse.so file to .joanju/proparse.dll.

ProRefactor Configuration Files

Analyst makes use of ProRefactor project configuration files. A small set of ABL/4GL scripts can be found at www.oehive.org/prorefactor/proj_cfg_dump for generating the files and directory structure that must go into the web application directory. Those need to be copied into your web application directory, so you have something like this:

  TOMCAT/webapps/analyst/prorefactor/projects/yourproject/*.properties

Edit your project's proparse.properties, and be sure that schema_file uses a fully qualified reference to your schema file. Tomcat must have read permissions for that file. It can be anywhere on your file system.

Server Properties File

rootDirs=/work/yourproject,/work2/thirdparty
prorefactorProject=yourproject

The rootDirs property is a comma-delimited list of root directories where your application source code can be found. Analyst uses this for finding all program files for indexing. Either use forward slashes or else remember to double your backslashes to escape them.

The prorefactorProject property is the project name that you used in the previous (ProRefactor configuration) step.

MySQL

Early releases of Analyst used an embedded database, allowing for MySQL as an optional external database. MySQL is no longer optional, but the configuration and setup has not yet been streamlined (to be done soon).

You point Analyst at a MySQL database by setting up a few files in the .joanju directory.

IMPORTANT: Analyst expects to start with an empty database. When it finds that there are no tables, it creates them, along with some important seed data. So, do not create or copy the table definitions by hand.

Getting MySQL

On Linux, use the latest stable release as provided by your Linux distributor. For Windows, download and install the latest stable release. For example, as of March 2008, 5.0 appears to be the latest stable release and you might install the "Windows Essentials" package from: http://dev.mysql.com/downloads/mysql/5.0.html#win32

cg.hibernate.cfg.xml

The file cg.hibernate.cfg.xml is the Hibernate configuration file, where you put your database connection settings. Copy this as an example.

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
  "-//Hibernate/Hibernate Configuration DTD//EN"
  "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
  <session-factory>
    <property name="dialect">org.hibernate.dialect.MySQLDialect</property>
    <property name="connection.driver_class">com.mysql.jdbc.Driver</property>
    <property name="connection.url">jdbc:mysql://localhost/joanjucg</property>
    <property name="connection.username">joanjucg</property>
    <property name="connection.password">joanjucg123</property>
    <property name="c3p0.min_size">3</property>
    <property name="current_session_context_class">thread</property>
  </session-factory>
</hibernate-configuration>

callgraph.sql

The file callgraph.sql is the SQL script which is called to add or re-add the database tables at first run or after Clear Database is done. The semi-colon ";" must be the last character on the line for it to be found as the SQL statement separator. A double-dash "--" at the beginning of a line makes that line a comment.

CREATE TABLE bytecode (
	cu INTEGER, address INTEGER, opcode SMALLINT, data BLOB
	, INDEX bytecode_idx1 (cu, address)
	);

CREATE TABLE cufile (
	lowerstringid INTEGER, canonicalname TINYBLOB
	, INDEX cufile_idx1 (lowerstringid)
	, INDEX cufile_idx2 (canonicalname(64))
	);

CREATE TABLE lstring (
	id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY
	, lstring BLOB
	, INDEX lstring_idx1 (lstring(128))
	);

CREATE TABLE string (
	id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY
	, string TINYBLOB
	, INDEX string_idx1 (string(64))
	);

CREATE TABLE xref (
	srccu INTEGER, srcaddr INTEGER, tgtcu INTEGER, tgtaddr INTEGER
	, INDEX xref_idx1 (srccu, srcaddr)
	, INDEX xref_idx2 (tgtcu, tgtaddr)
	);

callgraphdrop.sql

The file callgraphdrop.sql is the SQL script which is called when Clear Database is done. See the previous section for script formatting rules.

DROP TABLE IF EXISTS bytecode;
DROP TABLE IF EXISTS cufile;
DROP TABLE IF EXISTS lstring;
DROP TABLE IF EXISTS string;
DROP TABLE IF EXISTS xref;

Creating a MySQL Database

Here is an example session for creating a database in MySQL for use by Analyst.

$ sudo mysql
CREATE DATABASE joanjucg;
CREATE USER joanjucg@localhost IDENTIFIED BY 'joanjucg123';
GRANT ALL ON joanjucg.* TO joanjucg@localhost;
quit

Reviewing the Administrative Page

In your web browser, on the Joanju.Analyst front page, you can find a link to the Administrative Page. There you can review the Server Information to check that Analyst is loading your configuration settings correctly.

If you have configured your own database connection (see Using an External Database), then click the Clear Database button to test that your database connection works OK.

Build/Refresh

From the Joanju.Analyst front page there is a link to the Build Manager. A build must be run before you will be able to browse or search. Submitting '/' by itself as a build job will build your entire application (as defined by your rootDirs setting), which may take a long time depending on the size of your application.

Before you launch a build though, you might want to consider some further configuration options, such as using an external database or some additional configuration scripts for injecting extra information into the call graph.

Configuration Scripts

Analyst uses callout hooks to script files so that you can augment its behavior and configuration. The scripts are written using the Java based scripting language Groovy.

These .groovy script files must go into a cgscripts subdirectory in your .joanju directory.

Note: The script file names are case sensitive.

Have a look at www.oehive.org/prorefactor/scripting/ for some examples of using Groovy scripting with ProRefactor, since much of that is applicable to scripting for Analyst. There you can find and download the latest TreeUtils.groovy, which is used in the example CallScript.groovy below. Those utility .groovy script files can be placed right in your cgscripts directory.

CallScript.groovy

The CallScript.groovy file is your own script for telling Analyst how to resolve calls that it otherwise would not resolve.

The CallScript.groovy class script must conform to com.joanju.cg.api.CallScriptI. Here is an example:

import com.joanju.cg.api.CallScriptReturn
import org.prorefactor.core.JPNode
import org.prorefactor.core.TokenTypes as T

/** User script for supplementing Analyst's call resolution.
 * This class is instantiated once for each compile unit when it's built.
 * This object's go() method will be run once for each call in the compile unit.
 */
class CallScript implements com.joanju.cg.api.CallScriptI {

    def matchSpec =
    [
    T.RUN, [
      T.VALUE, [
        T.LEFTPAREN,
        T.Widget_ref, [
          T.THISPROCEDURE,
          T.OBJCOLON,
          { it.getText().equalsIgnoreCase('PRIVATE-DATA')} ],
        T.RIGHTPAREN ],
      T.PERIOD ]
    ]

    /** This is the method that is run by Analyst when it is building a CU.
     * This method must return null or a CallScriptReturn object.
     */
    CallScriptReturn go(JPNode node) {
        
        if ( node.getFilename() ==~ /.*scripting.p$/ ) {
            def ret = new CallScriptReturn()
            
            if (TreeUtils.matchNode(node, matchSpec) ) {
                ret.addProcedureName('abc')
                return ret
            }
            
            if (node.firstChild()?.getText()?.equalsIgnoreCase('whatever')) {
                ret.addHandleValue('data/general/scripting2.p')
                return ret
            }
        }
        // If we got here, we let Analyst deal with it.
        return null
    }
}

AddSuperScript.groovy

The AddSuperScript.groovy file is your own script for telling Analyst how to resolve calls to ADD-SUPER that it otherwise would not resolve.

The AddSuperScript.groovy class script must conform to com.joanju.cg.api.AddSuperScriptI. Here is a simple example:

import com.joanju.cg.api.AddSuperScriptReturn
import org.prorefactor.core.JPNode

class AddSuperScript implements com.joanju.cg.api.AddSuperScriptI {

    AddSuperScriptReturn go(JPNode node) {
        AddSuperScriptReturn ret = null
        if (    node.getFilename() ==~ /.*example.p$/
            &&  node.firstNaturalChild().text.equalsIgnoreCase('super-hdl')
            ) {
            ret = new AddSuperScriptReturn()
            ret.addHandleValue('supers/super1.p')
            ret.addHandleValue('supers/super2.p')
        }
        return ret;
    }

}

BuildAugmenter.groovy

This hook just provides raw access to the application data at a few specific stages during the build process. It should be used with assistance from Joanju.

LuceneScript.groovy

The LuceneScript.groovy file is your own script for adding additional information to the Lucene search index.

The LuceneScript.groovy class script must conform to com.joanju.cgs.lucene.LuceneScriptI. Here is a silly example:

import org.apache.lucene.document.Document
import org.apache.lucene.document.Field
import org.prorefactor.treeparser.ParseUnit

class LuceneScript implements com.joanju.cgs.lucene.LuceneScriptI {
    
    void go(ParseUnit pu, Document doc) {
        def randomText = randomWord() + ' ' + randomWord()
        doc.add(new Field(
            'z_test', randomText, Field.Store.YES, Field.Index.TOKENIZED))
    }
    
    def randomWord() {
        def words = ['foo', 'bar']
        return words[(Math.random() * 2).toInteger()]
    }
    
}

cufilenames.groovy

See examples/cufilenames.groovy. This script gives you full control over the list of filenames which are treated as compile unit source files. For example, you might use this to add non-traditional CU filename extensions.

Unresolved Calls Report

Each call in the application is examined, one by one, to determine whether it can be fully resolved or not.

There are different reasons why a call might not be resolved. For example, the expression for a RUN VALUE statement may not be resolved by Analyst, or the handle for RUN IN handle may not be resolved. It is also possible that the target program may not have been found on the propath. All of these are listed in the report.

Why is this report useful? Resolving all calls is the first necessary step for certain types of modeling and re-engineering projects. If the goal is to ensure that all calls can be fully resolved, then this report serves as a checklist.

This report should also serve as a checklist when writing configuration scripts for Analyst to augment its own internal call resolution.

Mismatched Parameters Report

Each procedure in the application is examined and all calls to those procedures are checked for mismatched parameters.

Method Signatures

Analyst's method signatures for calls and procedures are displayed on the report, and they are also available from the public API. This section provides some details about those signatures.

methodsig: (param)* ;
param:
  ('B'|'I'|'O'|'X')
  ( buffertable
  | (bind)? datatype (extent)?
  | (bind)? temptable
  | (bind)? dataset
  ) ;
buffertable: ldbname '.' tablename ',' ;
temptable: 'T' (datatype (extent)? )* ',' ;
dataset: 'D' '(' temptable (temptable)* ')' ;
extent: '[' number ']' ;
datatype:
  'a'=clob, 'b'=blob, 'c'=char, class, 'd'=decimal, 'e'=comhandle, 'f'=date,
  'g'=datetime, 'h'=handle, 'i'=int, 'j'=datetimetz, 'k'=longchar, 'l'=logical,
  'm'=memptr, 'p'=raw, 'q'=recid, 'r'=rowid, 'u'=unknown, 'z'=other
  // NOTE: widget-handle is just 'h' for handle.
  ;
class: 'L' fullyQualClassName ',' ;
bind: 'N' ;

Notes

• RETURN from a DLL PROCEDURE EXTERNAL has to match to be matched with an OUTPUT, so 'O' is used rather than putting an 'R' into the sig.
• BIND is possible on a simple HANDLE parameter if it is a TEMP-TABLE or DATASET handle.
• Names (db.table and package.class) are all lowercase.
• Datatype 'z' "other" is mostly for the datatypes for DLLs, which are of no interest to Analyst.
• There is limited support for checking EXTENT. If the EXTENT is indeterminate, or if the extent is not defined with a hard-coded numeric literal (ex: EXTENT 12), then no further checking is done, and the extent is recorded as -1. This -1 value is to be read as "some extent, any extent" and is considered an OK match for any extent other than zero.
• There is limited support for datatype checking. Analyst checks the datatype of a variable that is passed as an argument, but it does not check the datatype of any other expressions. (There is no datatype propagation in the system.) Ex:

  	def var i1 as integer.
  	run p1 (input i1).  /* datatype of i1 is checked. */
  	run p1 (input i1 + 2).  /* datatype of expression is not checked. */
  	

  In these cases, 'u' is used for unknown datatype. Even though the datatype may not be fully checked, the direction (INPUT) is checked, and the fact that it is a simple parameter (rather than a TEMP-TABLE or DATASET parameter) is also checked.
• Analyst does not include the return datatype as part of the method signature.

Name Propagation Across Expressions

When propagating program names (string constants), Analyst will do some limited const propagation across string concatenation and SUBSTRING expressions. Some of these details may be useful if you want to know when and why Analyst is able or unable to propagate strings and program names.

String concat has an overflow cap of 100 values. (Ex: (str1 + str2), if str1 and str2 each have 40 values, there are 1600 possible values (combinations).) When 100 combinations are found, Analyst sets the expression value to UNRESOLVED.

String concat and SUBSTRING instructions only get calculated once, due to the potential for feedback loops. If a recalc is requested of one of those instructions, then unresolved is added to the set of possible values. An exception to this rule is for expressions that are procedure names: RUN VALUE(expression) and DYNAMIC-FUNCTION(expression). Those don't have the potential for feedback loops, so value propagation and recalculations do work their way through those expressions.

SUBSTRING only works with integer literals. There is no numeric propagation (only string propagation) in Analyst.

Eclipse Plug-in Client

The Eclipse plug-in is a small client which accesses an Analyst server using XML over HTTP. The .jar file for the plug-in contains the source, which you are welcome to copy and use any way you like.

Java version: This plug-in requires Java 5 or later. If you are using OpenEdge Architect, simply remove the -vm argument from your OpenEdge Architect shortcut (assuming you have Java 5 or later installed on your computer).

Update Site: The Eclipse update site URL is:
   http://www.joanju.com/analyst/clientsite
(From the Eclipse menu, choose Help - Software Updates - Find and Install - Search for new features to install).

Web browser: Analyst only supports Firefox and Opera, so Analyst does not use Eclipse's internal browser. Configure your Eclipse external browser to be Firefox or Opera (or Default system Web browser if one of those is already your system default). From the Eclipse menu choose Window - Preferences - General - Web Browser.

Server URL: The server URL is configured per project, on the project properties page.

Project Directories: The project location must be similar to one of the rootDir entries for the Analyst server configuration setting. For example, if abc/def/ghi.p is a file's project relative path in Eclipse, then it must also be a valid path relative to one of the server rootDir entries.

View in browser: The plug-in presents a context menu for any selected .p, .w, or .cls file. From the context menu you may launch the Analyst web browser view for the compile unit. The same menu is present in the editor, and if you have selected the name of a procedure or function, then Analyst will attempt to jump to that section of code in the preprocessed code browse view.

Where called: From the context menu for a .p or .w program file, you can launch a Where-called search for all RUNs which call that compile unit. The same menu is present in the editor, and if you have selected the name of a procedure or function, then Analyst will perform a where-called search for that internal procedure or function.

Call targets: First select RUN in the editor, or select the text DYNAMIC-FUNCTION, or select the name of the function for a function call. Then, use the editor's pop-up context menu to launch the call targets search. If Analyst finds exactly one call target, it will attempt to open it in the editor, otherwise, the call targets search results are opened in the search view. For a function or procedure which is simply a redirect to one of those IN a super or another handle, you may highlight the text IN in the editor to perform the call targets search.

Call Graphing Limitations

The call graphing engine in Analyst is capable of basic string propagation for resolving most kinds of calls. That includes RUN VALUE, RUN IN handle, DYNAMIC-FUNCTION, SUPER, and calls which resolve to session or local super procedures. The following list is a summary of language elements which are not currently handled by the call graphing engine.

• String functions other than concatenation and SUBSTRING. See Name Propagation Across Expressions for details.
• search-target for super procedures
• publish/subscribe
• OO syntax for methods, constructors, or destructors
• triggers

Most of these were planned for as part of the original product design. Implementation is just a matter of priorities. As always, if any of these are important to you, please let me know.