Archive Page 2

__LINE__ __FILE__ macro for java. Part II.

Second part of this post

How does it work?

As the jvm loads each class, it passes it to each instrumentation which registers a ClassFileTransformers. This gives a chance to alter its byte code. I am using this system to remove the reference to the __LINE__ field, and change it with the current line.
Let’s see the byte code of a simple class, to see how we can manipulate it

System.err.println( __FILE_LINE__ );

This simple code is compiled to the following bytecode.

   0:   getstatic       #8; //Field java/lang/System.err:Ljava/io/PrintStream;
   3:   getstatic       #14; //Field com/dg/ab/use/instument/CodeLocation.__FILE
_LINE__:Ljava/lang/String;
   6:   invokevirtual   #20; //Method java/io/PrintStream.println:(Ljava/lang/String;)V 

This can be easily generated using javap –c, which is included in any jdk

At line 3, we can see the actual instruction, which is loading the content of the __FILE__ into the stack. At this point, if you can just change this instruction to load a string which contains the actual line position, we solve our problem.
So we expect to convert the previous bytecode into this one

   0:   getstatic       #8; //Field java/lang/System.err:Ljava/io/PrintStream;
   3:   ldc     #14; //String Test.java:7
   5:   invokevirtual   #16; //Method java/io/PrintStream.println:(Ljava/lang/St
ring;)V

And doing that is extremely easy using the ASM bytecode manipulation library.
We just have to load the class, and ask a ClassReader to process it.
During the process, the ClassReader will produce some visiting events, that will be passed to our ClassAdator which will slightly change the content, and pass the event to a ClassWriter that will write the visiting event to bytecode.

Schema
To come

Code

public class CodeLocationClassTransformer  implements ClassFileTransformer {
	@Override
	public byte[] transform(ClassLoader loader, String className, Class redefiningClass, ProtectionDomain domain, byte[] bytes) throws IllegalClassFormatException {
		System.out.println("CodeLocation to Transform Class: " + className);
		ClassReader cr = new ClassReader(bytes);
		ClassWriter cw = new ClassWriter(cr, 0);
		CodeLocationClassAdapter adapter = new CodeLocationClassAdapter(cw);
		cr.accept(adapter, 0);
	    byte[] result = cw.toByteArray();
		return result;
	}

	public class CodeLocationClassAdapter extends ClassAdapter {
		private String source;
		public CodeLocationClassAdapter(ClassVisitor cv) {
			super(cv);
		}
		
		@Override
		public void visitSource(String source, String debug) {
			this.source = source;
			cv.visitSource(source, debug);
		}

		@Override
		public MethodVisitor visitMethod(int access, String name, String desc,
				String signature, String[] exceptions) {
			MethodVisitor mv;
			mv = cv.visitMethod(access, name, desc, signature, exceptions);
			if (mv != null) {
				mv = new CodeLocationMethodAdapter(this, mv);
			}
			return mv;
		}
	}

	public class CodeLocationMethodAdapter extends MethodAdapter {
		private Integer lineSeen;
		private CodeLocationClassAdapter owner;
		public CodeLocationMethodAdapter(CodeLocationClassAdapter owner, MethodVisitor mv) {
			super(mv);
			this.owner = owner;
		}

		@Override
		public void visitFieldInsn(int opcode, String owner, String name, String desc) {
			if ( opcode == Opcodes.GETSTATIC ) {
				// replace the instruction 
				// getstatic       #23; //Field com/dg/ab/use/instument/CodeLocation.__FILE_LINE__:Ljava/lang/String;
				if ( owner.equals(CodeLocation.class.getName().replace(".", "/"))) {
					if ( name.equals("__LINE__" ) ) {
						// by
						// ldc #xx // "linenumber"
						super.visitLdcInsn(String.valueOf(lineSeen));
						return;
					} else 
					if ( name.equals("__FILE__" ) ) {
						// replace with a ldc
						// ldc #xx // "source file"
						super.visitLdcInsn(this.owner.source);
						return;
					} else 
					if ( name.equals("__FILE_LINE__" ) ) {
						// replace with a ldc
						// ldc #xx // "source file:linenumber"
						super.visitLdcInsn(this.owner.source + ":" + String.valueOf(lineSeen));
						return;
					} 
				}
			}

			super.visitFieldInsn(opcode, owner, name, desc);
		}
		
        public void visitLineNumber(int line, Label paramLabel) {
        	this.lineSeen = line; 
        	super.visitLineNumber( line, paramLabel );
        }
	}

}

As you can see there are 3 main steps used during the byte code manipulation.

  • CodeLocationClassTransformer

    Class call by the jvm, in order to instument the classes. Transform the class by processing the bytecode through a CodeLocationClassAdapter
  • CodeLocationClassAdapter

    Will That store the current filename and process the Method bytecode through a CodeLocationMethodAdaptor.
  • CodeLocationMethodAdapter

    store the current line information, as provided by the ASM library. And convert during the visiting instruction events, the GETSTATIC on our fields ( __LINE__, __FILE__ ) to the LDC instruction with the appropriate parameters.

The only thing which is still missing is the code to register the CodeLocationClassTransformer.
This is done in the class PremainCodeLocaiton

public class PremainCodeLocation {
	public static void premain(String agentArguments, Instrumentation instrumentation) {	
		instrumentation.addTransformer(new CodeLocationClassTransformer());
	}
}

The agent premain class is called whe the agent jar is provided in the command line of the java, and this jar contains the appropriate entry in the manifest.

Manifest-Version: 1.0 
Premain-Class: com.dg.ab.use.instument.PremainCodeLocation

Vmware server 2.0, blank page on login

I was getting a blank page while trying to reach my vmware server web ui, with some really strange java script error, since few days.
I was also getting some error page in the log file of vmare-mui ( /var/log/vmware-mui )

[error] Failed to connect. Reason = (11) 501 Global command GLOBAL server-vmdb to non-host agent targets not supported.\n

After upgrading a new version of vmware server (2.0.2), the problem was still occurring. Really frustrating !

I finally made it working.

Basically it is an incompatibility between the web ui of vmware server during the ssl 3.0 handshake, which is crashing the server.
The javascript errors was coming from partially loaded files.And due to the watchdog which is protecting the webserver, I never really notice that the server was crashing really often !

The solution is to activate the ssl 2.0 on your firefox. To do so, put about:config in the url, pass the warning page.
And activate the ssl 2.0 by changing the flags

security.enable_ssl2 : true
security.enable_ssl3 : false

about:config

__LINE__ __FILE__ macro for java. Instrumentation, Bytecode manipulation ab’USE

While driving back home, I was thinking about a use for the Instrumentation of java, and I thought of implementing the __LINE__ macro like in c.

Instrumentation what is that ? An instrumentation service is a way of modifying the loaded java bytes code on the fly, before it really exists in the jvm. The main usages of services are to implements memory profiling, code profiling and most ( all ? ) AOP frameworks.

__LINE__ __FILE__ for java

As you know, as a java developer you do not have access to a preprocessor. Most of the time, you can live with it. But when it comes to logging, it is another thing. If you are like me, most of the time you want to print with your log statement, with the classname/file name which issued the log statement and also want to see the line number.
Good logging framework provides this, but how it is fetching the information is amazingly slow.
When the logging framework needs to locate the location of the log statement, it needs to get a snapshoot of the calling stack, then needs to guess who is the caller, by carefully examining it.

Let’s examine the code which does this in the logging framework provided with the jdk ( java.util.logging.* )

// Private method to infer the caller's class and method names
private void inferCaller() {
	needToInferCaller = false;
	// Get the stack trace.
	StackTraceElement stack[] = (new Throwable()).getStackTrace();
	// First, search back to a method in the Logger class.
	int ix = 0;
	while (ix < stack.length) {
	    StackTraceElement frame = stack[ix];
	    String cname = frame.getClassName();
	    if (cname.equals("java.util.logging.Logger")) {
		break;
	    }
	    ix++;
	}
	// Now search for the first frame before the "Logger" class.
	while (ix < stack.length) {
	    StackTraceElement frame = stack[ix];
	    String cname = frame.getClassName();
	    if (!cname.equals("java.util.logging.Logger")) {			 
		// We've found the relevant frame.
	        setSourceClassName(cname);
	        setSourceMethodName(frame.getMethodName());
		  // I do not understand why they do not try to load the getLineNumber 
		return;
	    }
	    ix++;
	}
	// We haven't found a suitable frame, so just punt.  This is
        // OK as we are only committed to making a "best effort" here.
    }

A lot of code just to detect the caller, isn'it ? And even more it does not work all the time! In the following example, the line number will be improperly guessed!

public class Test {
  Logger logger = Logger.getLogger( Test.class.getName() );
  public void main() {
    error( “Error in main” );  					<-- I want this line
  }

  private void error( String message ) {
    logger.sever( message ) 						<-- I wll be getting this one
  }
}

You are basically not allowed to pass by any other method before logging!

My solution

Is to define some static fields which will be returning the __FILE__ ...
And behind the scene I use the instrumentation, byte code manipulation, and the available debug information to dynamically add the correct file/line number.

Here is how to use it

package com.dg.test;

import static com.dg.ab.use.instument.CodeLocation.__FILE__;
import static com.dg.ab.use.instument.CodeLocation.__LINE__;
import static com.dg.ab.use.instument.CodeLocation.__FILE_LINE__;

public class CodeLocationTest {
	public void run() {
		System.err.println( __FILE_LINE__ );
		System.err.println( __FILE_LINE__ + ": Entering in run method" );
		System.err.println( __FILE_LINE__ + ": Entering in run method" );
		for ( int i = 0; i < 10; i++ ) {
			System.err.println( __FILE__ + ":" + __LINE__ + " value of i is '"+i+"'");
		}
	}
	public static void main( String args[] ) {
		new Test().run();
	}
}

Thanks to the static import, it really looks like c code isn’t it ? And when running it with the appropriate agent ( adding -javaagent:CodeLocationAgent.jar to you java parameter), the __LINE__, __FILE__, __FILE_LINE__ will be properly replaced, and give this result !

CodeLocationTest.java:9
CodeLocationTest.java:10: Entering in run method
CodeLocationTest.java:11: Entering in run method
CodeLocationTest.java:13 value of i is '0'
CodeLocationTest.java:13 value of i is '1'
CodeLocationTest.java:13 value of i is '2'
CodeLocationTest.java:13 value of i is '3'
CodeLocationTest.java:13 value of i is '4'
CodeLocationTest.java:13 value of i is '5'
CodeLocationTest.java:13 value of i is '6'
CodeLocationTest.java:13 value of i is '7'
CodeLocationTest.java:13 value of i is '8'
CodeLocationTest.java:13 value of i is '9'

Want to try ? Download the code from my svn repository, and run the Test.launch run configuration.

If you are sceptic about the performance boost, I even made a comparison of both systems.
Just run the testcase Speedtest, which is running about 1OOO times the same test, but disable the output stream to remove the lag of logging to the console.
The result ?

By inferring the location : 400 ms
Using the CodeLocationAgent : 50 ms

But of course without counting time took to process the class bytecode.

Next week I will explain you how I did it easily, with the help of the ASM library!
But in the meantime, you can check the source code on the svn.

Sources :

The agent code svn repository is http://svn.gallot.be/blog/Instument-abUse/Instrument-abUse/
The test code svn repository is http://svn.gallot.be/blog/Instument-abUse/Instrument-abUse-tets/

SquidGuard, Content Filtering, Password bypass integrated with a windows domain

At home, I installed SquidGuard in order to protect my children againt suspicious sites. For that purpose I just installed squid as a transparent proxy on my linux router and configured SquidGuard using some well known blacklist rules. This setup was running quite well, with only one small problem, I had no way to bypass the system, when the rules were a little bit too strict.

I wanted to show a login page instead of an error page. And by login on the page, using my windows username/password (checked against my windows domain), be able to bypass the blocking rules.
I searched on the internet for a solution, but I did not really find one which was working with a transparent proxy. So I developed mine :)

Note: If you want to re-use my solution, you first need a working version of squid and squidguard. There are a lot of tutorials which will guide you to do that. You also require to have a working configuration of winbind, that is to say that your samba installation needs to be integrated with you windows domain ( wbinfo –a USER%PASS must work ! ).

Here is my solution.

Squidguard configuration and related pages

First, when the destination matches the rules, I redirect to a login page instead of the usual error page.

Excerpt from /etc/squid/squidGuard.conf

dest adult {
       domainlist      adult/domains
       urllist         adult/urls
       expressionlist  adult/expressions
       redirect         https://squid.dgconsulting.local/adultcheck/login.php?rurl=%u
}

This login page is asking the user and password of the user. And it useq winbind to check if the user/password is correct. If it is correct, it adds the client ip to the ip database with the current date.

Https is mandatory since the password is passed as clear text within the http request

File: /var/www/adultcheck/login.php



Adult check

Username:
Password:
Url:
quote($user).", ".$db->quote($ip).", CURRENT_TIMESTAMP + INTERVAL ".$db->quote($time)." MINUTE )"; $result = $db->query( $query ); return $result; } } $ip = getIP(); $username=$_POST['username']; $password=$_POST['password']; $authtime=30; if ( isset( $username ) ) { $return_var = 0; $output = array(); exec('/usr/bin/wbinfo -K '.$username.'%'.$password, $output, $return_var); if ( $return_var == 0 ) { echo $ip." Authoriser pour ".$authtime." minutes
\n"; $db = DB::connect('mysql://adultcheck:adultcheck@localhost/adultcheck'); $db->setFetchMode(DB_FETCHMODE_ASSOC); AdultHost::authorize($username,$ip,$authtime); echo "continuer
"; echo " \n "; } else { echo "Password Invalide
\n"; } } ?>

Squid configuration and related tools

Secondly, I need to tell squid in which case the rules apply, and in which case they must be bypassed. The configuration option external_acl_type allows me to do that. I can receive the ip of the http client, check if the ip has recently been added in the database, and either I allow to go directly or through the filter if the ip is not found.

Here is the relevant configuration I my squig.conf

Excerpt from /var/www/adultcheck/squid_acl.php

acl adults external checkip
external_acl_type checkip ttl=5 negative_ttl=5 %SRC /var/www/adultcheck/squid_acl.php
url_rewrite_access deny adults

At each request, the ip of the requestor is checked against the database. If the IP is found in the database, the request is marked as adult. Any request which is marked as adult does not get into the url redirector ( squigguard )

Here is the other relevant code

File: /var/www/adultcheck/squid_acl.php

#!/usr/bin/php
setFetchMode(DB_FETCHMODE_ASSOC);

class AdultHost
{
    function getId($ip)
    {
        global $db;
        $query = "SELECT ID FROM ADULT_HOST where IP=".$db->quote($ip)." and END > CURRENT_TIMESTAMP ";
        $result = $db->getOne( $query );
        return $result;
    }
}

class logfile
{
  function info($the_string )
  {
    $this->write( "I: ".$the_string."\n" );
  }

  function write($the_string )
  {
    if( $fh = @fopen( "/tmp/adultcheck.log", "a+" ) )
    {
      fputs( $fh, $the_string, strlen($the_string) );
      fclose( $fh );
      return( true );
    }
    else
    {
      return( false );
    }
  }
}

  $log = new logfile();

  set_time_limit(0);
  define('STDIN',fopen("php://stdin","r"));
  while (!0)
  {
    $IP = trim(fgets(STDIN));
    if (feof(STDIN)) {
      break;
    }
    $log->info( "IP: $IP" );
    $id = AdultHost::getId( $IP );
    if ( isset( $id ) ) {
      echo ( "OK\n" );
    } else
    {
      echo ( "ERR\n" );
    }
  }

?>

The database table where the ip are stored

CREATE TABLE ADULT_HOST (
  ID int(11) NOT NULL auto_increment,
  USER varchar(20) NOT NULL,
  IP varchar(20) NOT NULL,
  START timestamp NOT NULL default CURRENT_TIMESTAMP,
  END timestamp NOT NULL default '0000-00-00 00:00:00',
  PRIMARY KEY  (ID)
)

Conclusion

This solution is working at home for more than 1 year now. It is working fine. The only small problem is that sometimes you get redirected twice to the login page.
This is due to the fact that squid is not really rechecking the acl at each request, but only each 5 seconds ( as defined in the ttl parameter of external_acl_type )
There are also other improvements that can be useful if not mandatory for a production installation.

  • Checking if the user against is member of a special windows group
  • The username and password are passed as parameters to the winbind application. This may cause a security breach. With the appropriate access to the server you may find the password. But to be honest, I already know the password of all users of my windows domain ! The solution would be to use NTLM authentication, but I do not want to spend too much time just for that.

RMI, JDK 1.6, OSGi ( 1 of 2 )

Introduction

If you ever wanted to search for some information about Rmi, you probably founds the sun tutorial which is located here : http://java.sun.com/docs/books/tutorial/rmi/overview.html

This tutorial is really complete, but it has been there for years now. And since then, some big improvements have been done on the rmi support of java. For example since 1.5 you do not have to create the stub or the skeleton class.

Due to that, I believe that there are easier ways to implement a RMI service now, and that is what I will try to show you in this article.

While preparing this small article, I digged a little bit on google. It seems that a lot of users have problems or at least had problems in the past, especially because the codebase classloading messes.
It is required to put your jars on a webserver, set a security manager, and set some system properties to find back the published jars, and only then the magic of rmi class loading will occur.
But does this fit on an osgi architecture, where each application has its own well defined dependency context, defined using a declarative way. And does the service support interface versionning ?
Basically no, especially for the interface versionning !

In this article, I will try to show you my way of implementing rmi application, in a OSGi environment. IMHO using the easier way, or at least a less messy way.

Writting the SimpleRmiServer

While trying to be as simple as I can, I will try to show you how to write a RMI Service that will use most of the main features of RMI :

  • Publishing a RMI interface
  • Sending custom object and receiving it back
  • And even show you how to implement a service callback

Design the remote interfaces

Sources : The svn repository is http://svn.gallot.be/blog/rmi-osgi-jvm16/part-1/SimpleRmiInterface/

The remote interface of my test service is really simple. It simply contains 2 methods

  • The first one is used to execute some codes on the server synchronously
  • The second method allows you to post a request, and to be called when the result is ready

public interface ISimpleRmiService extends Remote {
  public static final String BIND_NAME = "ISimpleRmiService";
  public Response request(Request request) throws RemoteException;
  public void asynchronousRequest(ICallback callback, Request request) throws RemoteException;
}

Remember that any RMI interface needs to follow some rules !

  • The interface must extend the interface java.rmi.Remote
  • Any methods of this interface must throw java.rmi.RemoteException
  • Any parameters of this interface must be serialisable
  • Any callback interface must follow these rules as well
  • And as a personal rule, I also add the binding name as a static field, and prefix the interface name with I

This is it ! As you can see the interface is composed of two methods

Let’s now design the callback interface using the same rules


public interface ICallback extends Remote {
  public void response( Response response ) throws RemoteException;
}

And the parameter object, request and response are basically a Serializable bean which is holding a string value.


public class Request implements Serializable {
  private static final long serialVersionUID = 1L;
  private String value;
  public Request( String value ) {
    this.value = value;
  }
  public String setValue() {
    return value;
  }
}

Then all these interfaces are putted in a single OSGi bundle, which is going to be shared between the client and server.

Implementing the remote server

Sources : The svn repository is http://svn.gallot.be/blog/rmi-osgi-jvm16/part-1/SimpleRmiServer/

The code of the remote service does 3 things.

Create the RMI Registry

The rmiregistry is a special component in the RMI architecture used to expose the published service.
The normal way of starting it, is to run the rmiregistry command. But personally I do not really like having to manually start the rmiregistry by hand. It is basically another process to monitor in the system, and even more, if for any reason the rmiregistry is restarted, every RMI server needs to be exported again!
But since the JDK 1.1, it is possible to start the rmiregistry using a simple call to LocateRegistry.createRegistry
I believe that this way of handling the rmiregisty is better most of the time, it is tying your rmi server with the rmiresitry within the same jvm. At least when you start your rmi server, the rmigistry starts automatically!
Also, having the rmiregistry and the rmi service inside the same jvm allows you to bypass some of the class loader messes of rmi.

Implementing the service class

The rmi service class is really easy to implement, it is a simple class implementing the remote interface, and extending UnicastRemoteObject.

public class SimpleRmiService extends UnicastRemoteObject implements ISimpleRmiService {
  public Response request(Request request) {
    Response response = new Response( "Response @ " + request.getValue());
    return response;
  }

  public void asynchronousRequest(final ICallback callback, final Request request) throws RemoteException {
    executor.execute( new Runnable() {
      public void run() {
        try {
          Response response = new Response( "Response @ " + request.getValue() );
          callback.response(response);
        } catch ( Exception ex ) {
          ex.printStackTrace();
        }
      }
    });
  }
}

And to publish it, simply register it with the remote registry previously created during the first step

Binding the Rmi service class

You can instantiate the rmi service class, and publish it to the rmiregistry.

One good place to start the rmi registry and to bind the rmi service is the Bundle Activator class of your bundle. In the start method of the bundle activator, simply do :

Registry registry = LocateRegistry.createRegistry( Registry.REGISTRY_PORT );
registry.bind( ISimpleRmiService.BIND_NAME, new SimpleRmiService());

To try the server part of this application, simply start your favourite osgi framework, and install the interface bundle and the service bundle. Or just right click on the “SimpleRmiServer.launch” file and run it.

Implementing the remote client

Sources : The svn repository is http://svn.gallot.be/blog/rmi-osgi-jvm16/part-1/SimpleRmiClient/

Implementing the client is even easier.

  • You just need to first connect on the rmi registry
  • Lookup the rmi service using the binding name
  • Call the method

Registry registry = LocateRegistry.getRegistry( Registry.REGISTRY_PORT );
ISimpleRmiService simpleRmiService = (ISimpleRmiService)registry.lookup( ISimpleRmiService.BIND_NAME );
Response response = simpleRmiService.request( request );

But what about the callback implementation ?

It is almost the same method as for the ‘direct’ call, you just need to provide a correct ICallback implementation.
The ICallback implementation is just another rmi service binded anonymously, therefor it needs to behave appropriately as a remote object.
As for the main service, the best way to achieve this is to extend the UnicastRemoteObject.


public class Callback extends UnicastRemoteObject implements ICallback {
  private static final long serialVersionUID = 1L;
  public Callback() throws RemoteException {
  }
  public void response( Response response ) {
    System.out.println("Response received: "+response);
    System.out.println("Response : "+response.getClass().getClassLoader());
  }
}

To test it, just run the run configuration named ‘SimpleRmiClient.launch’

Summary

As you can see, it is working. No need to create the stub using the rmic tools, no need to start the rmi registry, and you do not need to publish the jar interface on a webserver.

However this solution is too light for a real life example.

  • You cannot have multiple rmi services, since the registry is started with the SimpleRmiServer
  • Restarting the SimpleRmiServer will fail, when the registry is rebinded



These problems will be solved in my next article, and I will even try to show you how to publish different versions of the same interface !