March 23rd, 2010 by Dominique Gallot
While driving back home, I was thinking about a use for the Instrumentation of java, and I thought of implementing the __LINE__ macro like in c.
Instrumentation what is that ? An instrumentation service is a way of modifying the loaded java bytes code on the fly, before it really exists in the jvm. The main usages of services are to implements memory profiling, code profiling and most ( all ? ) AOP frameworks.
__LINE__ __FILE__ for java
As you know, as a java developer you do not have access to a preprocessor. Most of the time, you can live with it. But when it comes to logging, it is another thing. If you are like me, most of the time you want to print with your log statement, with the classname/file name which issued the log statement and also want to see the line number.
Good logging framework provides this, but how it is fetching the information is amazingly slow.
When the logging framework needs to locate the location of the log statement, it needs to get a snapshoot of the calling stack, then needs to guess who is the caller, by carefully examining it.
Let’s examine the code which does this in the logging framework provided with the jdk ( java.util.logging.* )
// Private method to infer the caller's class and method names
private void inferCaller() {
needToInferCaller = false;
// Get the stack trace.
StackTraceElement stack[] = (new Throwable()).getStackTrace();
// First, search back to a method in the Logger class.
int ix = 0;
while (ix < stack.length) {
StackTraceElement frame = stack[ix];
String cname = frame.getClassName();
if (cname.equals("java.util.logging.Logger")) {
break;
}
ix++;
}
// Now search for the first frame before the "Logger" class.
while (ix < stack.length) {
StackTraceElement frame = stack[ix];
String cname = frame.getClassName();
if (!cname.equals("java.util.logging.Logger")) {
// We've found the relevant frame.
setSourceClassName(cname);
setSourceMethodName(frame.getMethodName());
// I do not understand why they do not try to load the getLineNumber
return;
}
ix++;
}
// We haven't found a suitable frame, so just punt. This is
// OK as we are only committed to making a "best effort" here.
}
A lot of code just to detect the caller, isn'it ? And even more it does not work all the time! In the following example, the line number will be improperly guessed!
public class Test {
Logger logger = Logger.getLogger( Test.class.getName() );
public void main() {
error( “Error in main” ); <-- I want this line
}
private void error( String message ) {
logger.sever( message ) <-- I wll be getting this one
}
}
You are basically not allowed to pass by any other method before logging!
My solution
Is to define some static fields which will be returning the __FILE__ ...
And behind the scene I use the instrumentation, byte code manipulation, and the available debug information to dynamically add the correct file/line number.
Here is how to use it
package com.dg.test;
import static com.dg.ab.use.instument.CodeLocation.__FILE__;
import static com.dg.ab.use.instument.CodeLocation.__LINE__;
import static com.dg.ab.use.instument.CodeLocation.__FILE_LINE__;
public class CodeLocationTest {
public void run() {
System.err.println( __FILE_LINE__ );
System.err.println( __FILE_LINE__ + ": Entering in run method" );
System.err.println( __FILE_LINE__ + ": Entering in run method" );
for ( int i = 0; i < 10; i++ ) {
System.err.println( __FILE__ + ":" + __LINE__ + " value of i is '"+i+"'");
}
}
public static void main( String args[] ) {
new Test().run();
}
}
Thanks to the static import, it really looks like c code isn’t it ? And when running it with the appropriate agent ( adding -javaagent:CodeLocationAgent.jar to you java parameter), the __LINE__, __FILE__, __FILE_LINE__ will be properly replaced, and give this result !
CodeLocationTest.java:9
CodeLocationTest.java:10: Entering in run method
CodeLocationTest.java:11: Entering in run method
CodeLocationTest.java:13 value of i is '0'
CodeLocationTest.java:13 value of i is '1'
CodeLocationTest.java:13 value of i is '2'
CodeLocationTest.java:13 value of i is '3'
CodeLocationTest.java:13 value of i is '4'
CodeLocationTest.java:13 value of i is '5'
CodeLocationTest.java:13 value of i is '6'
CodeLocationTest.java:13 value of i is '7'
CodeLocationTest.java:13 value of i is '8'
CodeLocationTest.java:13 value of i is '9'
Want to try ? Download the code from my svn repository, and run the Test.launch run configuration.
If you are sceptic about the performance boost, I even made a comparison of both systems.
Just run the testcase Speedtest, which is running about 1OOO times the same test, but disable the output stream to remove the lag of logging to the console.
The result ?
By inferring the location : 400 ms
Using the CodeLocationAgent : 50 ms
But of course without counting time took to process the class bytecode.
Next week I will explain you how I did it easily, with the help of the ASM library!
But in the meantime, you can check the source code on the svn.
Sources :
The agent code svn repository is http://svn.gallot.be/blog/Instument-abUse/Instrument-abUse/
The test code svn repository is http://svn.gallot.be/blog/Instument-abUse/Instrument-abUse-tets/
March 17th, 2010 by Dominique Gallot
At home, I installed SquidGuard in order to protect my children againt suspicious sites. For that purpose I just installed squid as a transparent proxy on my linux router and configured SquidGuard using some well known blacklist rules. This setup was running quite well, with only one small problem, I had no way to bypass the system, when the rules were a little bit too strict.
I wanted to show a login page instead of an error page. And by login on the page, using my windows username/password (checked against my windows domain), be able to bypass the blocking rules.
I searched on the internet for a solution, but I did not really find one which was working with a transparent proxy. So I developed mine 
Note: If you want to re-use my solution, you first need a working version of squid and squidguard. There are a lot of tutorials which will guide you to do that. You also require to have a working configuration of winbind, that is to say that your samba installation needs to be integrated with you windows domain ( wbinfo –a USER%PASS must work ! ).
Here is my solution.
Squidguard configuration and related pages
First, when the destination matches the rules, I redirect to a login page instead of the usual error page.
Excerpt from /etc/squid/squidGuard.conf
dest adult {
domainlist adult/domains
urllist adult/urls
expressionlist adult/expressions
redirect https://squid.dgconsulting.local/adultcheck/login.php?rurl=%u
}
This login page is asking the user and password of the user. And it useq winbind to check if the user/password is correct. If it is correct, it adds the client ip to the ip database with the current date.
Https is mandatory since the password is passed as clear text within the http request
File: /var/www/adultcheck/login.php
Adult check
quote($user).", ".$db->quote($ip).", CURRENT_TIMESTAMP + INTERVAL ".$db->quote($time)." MINUTE )";
$result = $db->query( $query );
return $result;
}
}
$ip = getIP();
$username=$_POST['username'];
$password=$_POST['password'];
$authtime=30;
if ( isset( $username ) ) {
$return_var = 0;
$output = array();
exec('/usr/bin/wbinfo -K '.$username.'%'.$password, $output, $return_var);
if ( $return_var == 0 ) {
echo $ip." Authoriser pour ".$authtime." minutes
\n";
$db = DB::connect('mysql://adultcheck:adultcheck@localhost/adultcheck');
$db->setFetchMode(DB_FETCHMODE_ASSOC);
AdultHost::authorize($username,$ip,$authtime);
echo "
continuer";
echo " \n ";
} else
{
echo "Password Invalide
\n";
}
}
?>
Squid configuration and related tools
Secondly, I need to tell squid in which case the rules apply, and in which case they must be bypassed. The configuration option external_acl_type allows me to do that. I can receive the ip of the http client, check if the ip has recently been added in the database, and either I allow to go directly or through the filter if the ip is not found.
Here is the relevant configuration I my squig.conf
Excerpt from /var/www/adultcheck/squid_acl.php
acl adults external checkip
external_acl_type checkip ttl=5 negative_ttl=5 %SRC /var/www/adultcheck/squid_acl.php
url_rewrite_access deny adults
At each request, the ip of the requestor is checked against the database. If the IP is found in the database, the request is marked as adult. Any request which is marked as adult does not get into the url redirector ( squigguard )
Here is the other relevant code
File: /var/www/adultcheck/squid_acl.php
#!/usr/bin/php
setFetchMode(DB_FETCHMODE_ASSOC);
class AdultHost
{
function getId($ip)
{
global $db;
$query = "SELECT ID FROM ADULT_HOST where IP=".$db->quote($ip)." and END > CURRENT_TIMESTAMP ";
$result = $db->getOne( $query );
return $result;
}
}
class logfile
{
function info($the_string )
{
$this->write( "I: ".$the_string."\n" );
}
function write($the_string )
{
if( $fh = @fopen( "/tmp/adultcheck.log", "a+" ) )
{
fputs( $fh, $the_string, strlen($the_string) );
fclose( $fh );
return( true );
}
else
{
return( false );
}
}
}
$log = new logfile();
set_time_limit(0);
define('STDIN',fopen("php://stdin","r"));
while (!0)
{
$IP = trim(fgets(STDIN));
if (feof(STDIN)) {
break;
}
$log->info( "IP: $IP" );
$id = AdultHost::getId( $IP );
if ( isset( $id ) ) {
echo ( "OK\n" );
} else
{
echo ( "ERR\n" );
}
}
?>
The database table where the ip are stored
CREATE TABLE ADULT_HOST (
ID int(11) NOT NULL auto_increment,
USER varchar(20) NOT NULL,
IP varchar(20) NOT NULL,
START timestamp NOT NULL default CURRENT_TIMESTAMP,
END timestamp NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (ID)
)
Conclusion
This solution is working at home for more than 1 year now. It is working fine. The only small problem is that sometimes you get redirected twice to the login page.
This is due to the fact that squid is not really rechecking the acl at each request, but only each 5 seconds ( as defined in the ttl parameter of external_acl_type )
There are also other improvements that can be useful if not mandatory for a production installation.
- Checking if the user against is member of a special windows group
- The username and password are passed as parameters to the winbind application. This may cause a security breach. With the appropriate access to the server you may find the password. But to be honest, I already know the password of all users of my windows domain ! The solution would be to use NTLM authentication, but I do not want to spend too much time just for that.
March 15th, 2010 by Dominique Gallot
Introduction
If you ever wanted to search for some information about Rmi, you probably founds the sun tutorial which is located here : http://java.sun.com/docs/books/tutorial/rmi/overview.html
This tutorial is really complete, but it has been there for years now. And since then, some big improvements have been done on the rmi support of java. For example since 1.5 you do not have to create the stub or the skeleton class.
Due to that, I believe that there are easier ways to implement a RMI service now, and that is what I will try to show you in this article.
While preparing this small article, I digged a little bit on google. It seems that a lot of users have problems or at least had problems in the past, especially because the codebase classloading messes.
It is required to put your jars on a webserver, set a security manager, and set some system properties to find back the published jars, and only then the magic of rmi class loading will occur.
But does this fit on an osgi architecture, where each application has its own well defined dependency context, defined using a declarative way. And does the service support interface versionning ?
Basically no, especially for the interface versionning !
In this article, I will try to show you my way of implementing rmi application, in a OSGi environment. IMHO using the easier way, or at least a less messy way.
Writting the SimpleRmiServer
While trying to be as simple as I can, I will try to show you how to write a RMI Service that will use most of the main features of RMI :
- Publishing a RMI interface
- Sending custom object and receiving it back
- And even show you how to implement a service callback
Design the remote interfaces
Sources : The svn repository is http://svn.gallot.be/blog/rmi-osgi-jvm16/part-1/SimpleRmiInterface/
The remote interface of my test service is really simple. It simply contains 2 methods
- The first one is used to execute some codes on the server synchronously
- The second method allows you to post a request, and to be called when the result is ready
public interface ISimpleRmiService extends Remote {
public static final String BIND_NAME = "ISimpleRmiService";
public Response request(Request request) throws RemoteException;
public void asynchronousRequest(ICallback callback, Request request) throws RemoteException;
}
Remember that any RMI interface needs to follow some rules !
- The interface must extend the interface java.rmi.Remote
- Any methods of this interface must throw java.rmi.RemoteException
- Any parameters of this interface must be serialisable
- Any callback interface must follow these rules as well
- And as a personal rule, I also add the binding name as a static field, and prefix the interface name with I
This is it ! As you can see the interface is composed of two methods
Let’s now design the callback interface using the same rules
public interface ICallback extends Remote {
public void response( Response response ) throws RemoteException;
}
And the parameter object, request and response are basically a Serializable bean which is holding a string value.
public class Request implements Serializable {
private static final long serialVersionUID = 1L;
private String value;
public Request( String value ) {
this.value = value;
}
public String setValue() {
return value;
}
}
Then all these interfaces are putted in a single OSGi bundle, which is going to be shared between the client and server.
Implementing the remote server
Sources : The svn repository is http://svn.gallot.be/blog/rmi-osgi-jvm16/part-1/SimpleRmiServer/
The code of the remote service does 3 things.
Create the RMI Registry
The rmiregistry is a special component in the RMI architecture used to expose the published service.
The normal way of starting it, is to run the rmiregistry command. But personally I do not really like having to manually start the rmiregistry by hand. It is basically another process to monitor in the system, and even more, if for any reason the rmiregistry is restarted, every RMI server needs to be exported again!
But since the JDK 1.1, it is possible to start the rmiregistry using a simple call to LocateRegistry.createRegistry
I believe that this way of handling the rmiregisty is better most of the time, it is tying your rmi server with the rmiresitry within the same jvm. At least when you start your rmi server, the rmigistry starts automatically!
Also, having the rmiregistry and the rmi service inside the same jvm allows you to bypass some of the class loader messes of rmi.
Implementing the service class
The rmi service class is really easy to implement, it is a simple class implementing the remote interface, and extending UnicastRemoteObject.
public class SimpleRmiService extends UnicastRemoteObject implements ISimpleRmiService {
public Response request(Request request) {
Response response = new Response( "Response @ " + request.getValue());
return response;
}
public void asynchronousRequest(final ICallback callback, final Request request) throws RemoteException {
executor.execute( new Runnable() {
public void run() {
try {
Response response = new Response( "Response @ " + request.getValue() );
callback.response(response);
} catch ( Exception ex ) {
ex.printStackTrace();
}
}
});
}
}
And to publish it, simply register it with the remote registry previously created during the first step
Binding the Rmi service class
You can instantiate the rmi service class, and publish it to the rmiregistry.
One good place to start the rmi registry and to bind the rmi service is the Bundle Activator class of your bundle. In the start method of the bundle activator, simply do :
Registry registry = LocateRegistry.createRegistry( Registry.REGISTRY_PORT );
registry.bind( ISimpleRmiService.BIND_NAME, new SimpleRmiService());
To try the server part of this application, simply start your favourite osgi framework, and install the interface bundle and the service bundle. Or just right click on the “SimpleRmiServer.launch” file and run it.
Implementing the remote client
Sources : The svn repository is http://svn.gallot.be/blog/rmi-osgi-jvm16/part-1/SimpleRmiClient/
Implementing the client is even easier.
- You just need to first connect on the rmi registry
- Lookup the rmi service using the binding name
- Call the method
Registry registry = LocateRegistry.getRegistry( Registry.REGISTRY_PORT );
ISimpleRmiService simpleRmiService = (ISimpleRmiService)registry.lookup( ISimpleRmiService.BIND_NAME );
Response response = simpleRmiService.request( request );
But what about the callback implementation ?
It is almost the same method as for the ‘direct’ call, you just need to provide a correct ICallback implementation.
The ICallback implementation is just another rmi service binded anonymously, therefor it needs to behave appropriately as a remote object.
As for the main service, the best way to achieve this is to extend the UnicastRemoteObject.
public class Callback extends UnicastRemoteObject implements ICallback {
private static final long serialVersionUID = 1L;
public Callback() throws RemoteException {
}
public void response( Response response ) {
System.out.println("Response received: "+response);
System.out.println("Response : "+response.getClass().getClassLoader());
}
}
To test it, just run the run configuration named ‘SimpleRmiClient.launch’
Summary
As you can see, it is working. No need to create the stub using the rmic tools, no need to start the rmi registry, and you do not need to publish the jar interface on a webserver.
However this solution is too light for a real life example.
- You cannot have multiple rmi services, since the registry is started with the SimpleRmiServer
- Restarting the SimpleRmiServer will fail, when the registry is rebinded
These problems will be solved in my next article, and I will even try to show you how to publish different versions of the same interface !
