How to match a file path with include/exclude pattern ?

SelectorUtils knows!

SelectorUtils.matchPath( include/excludePattern, filePath ) determines whether or not file’s path (I usually use a canonical one) is matching an <include> or <exclude> pattern. Very comfortable!

<dependency>
    <groupId>org.codehaus.plexus</groupId>
    <artifactId>plexus-utils</artifactId>
    <version>1.5.15</version>
    <scope>compile</scope>
</dependency>

TrueZip – Not only annoying but buggy as well!

It appears TrueZip is not only unnecessary complex and annoying to use (to say the least..) – but it’s very buggy as well (at least, version 6.6 that is available through repo1) :( 

Now, I wouldn’t mind for any other library to have some bugs here and there – everyone’s code has them, of course. But this one was getting on my nerve for a long time already. I bet it was someone’s university project: all this unnecessary complexity, all this amateur API with tons of duplicate methods doing almost the same .. I mean, just viewing a ZIP file as a “virtual file system” – doesn’t it look like someone’s thesis to you ?! Come one, people usually pack and unpack them, you know. With occasional updates which I think are so rare that it’s much simpler to unpack the archive to temporal folder and pack it back (which is already happening behind the scenes anyway – see below)

Some things that I’ve discovered lately:

1. If you pack a folder that contains other ZIP archives (I had to pack a folder with two big ZIPs inside, each one a JDK distribution) – the library seems to be repacking them!

   Why ? 

   – I don’t know. Probably because it’s so smart thing to do.

   But how do I know that ?

   – Size of (supposedly repacked) ZIP files is changing – it is dropping down a bit inside new archive
   – There are lot’s of “tzp-ctrl.tmp” files created in my temp folder during the process
   – New  archives do not open on Linux any more but only on Windows

2. Build process can’t be terminated when “pack” is in process – it can only be done by killing “java.exe”. So sweet.

3. There were many times when TrueZip Maven plugin  silently did nothing and created no archive when <includes> were involved (which I re-checked many times, of course, but I won’t be surprised if TrueZip has it’s own logic for how <include> works, like where it’s base directory and the like). So I had to pack the whole folder without <includes> and then remove the files that I didn’t want. Yes, it is silly, no doubt about that.

   And, btw, I really don’t like when things go wrong silently (which is a plague in some projects – Ant and Velocity are first coming to mind, you have an undefined property but those two ignore it silently).

   Hey, TrueZip! If for some reason I have wrong <include> patterns (which I’m sure were correct, but .. nevermind) and no archive is actually created – why being so silent about that ? In my code an exception is thrown when a combination of include and exclude patterns returns an empty Collection (and exception is specifying all three: base directory, include patterns, exclude patterns). 

4. Not a bug but a pure headache: things go really wrong if you forget to make this umount() call after creating an archive. Like clicking an “Ok” button – it flushes all data to disk. There’s also an update() call as well but it “may retain some temporary files” (surprise, surprise!) so one shouldn’t really call it (which I did until noticed this subtle difference)

I do hope I won’t have to post about this library again.

Maven – Disabling (escaping) properties interpolation

It seems someone up there is taking a great care of me so that I have some kind of unique problem to solve on a daily basis. That’s Ok, I don’t mind, keep on doing that, whoever you are ..

Today I had this weird case with Maven’s properties interpolation. Interpolation in dynamic languages is a process of replacing ${project.build.directory} property (or expression) with it’s actual value. It happens on different levels: in Ant, Perl and Groovy it happens inside strings: “this is just an ${example}” (Groovy’s GStrings provide a complete EL inside ${…}, much like JSP EL, which is .. well, groovy!). In Maven, interpolation is mostly known as filtering (which is done by “maven-resources-plugin” via “maven-filtering”) and <properties> where one can define a property as <artifactsDir>${project.build.directory}</artifactsDir> (related issues: 1, 2, 3, 4).

Maven also interpolates variables that are specified in plugin configurations:

<properties>
    <somePath>${project.basedir}/../../</somePath>
</properties>
...
<configuration>
    <root>${somePath}</root>
</configuration>

Here Mojo “root” parameter will be assigned a value of ${somePath} property defined above which is project’s basedir + two directories up.

So far so good. Right now I’m working on a plugin that is capable of making replacements in files copied (similar to filtering but using inline “from” pattern and “to” replacement that do not involve any Maven properties, I think it’s more convenient in some cases):

 

<resource>
    <targetPath>${directory.to.copy.the.file.to}</targetPath>
    <file>${directory.to.take.the.file.from}/build.xml</file>
    <replace>
        <to>
            <![CDATA[
<project default="run">
    <target name="run">
        <exec executable="${basedir}/some.exe" spawn="true"/>
    </target>
</project>
            ]]>
        </to>
    </replace>
</resource>

Here <from> tag is missing which means that all content of a file copied will be replaced by the content of a <to> tag. Which is what I’m doing here – generating a new “build.xml” file that <exec>s an .exe file, residing in the same directory where “build.xml” is generated. Note ${basedir} – that’s an expression I want to appear in generated “build.xml”. Unfortunately, Maven replaces it with value of ${project.basedir} which is POM’s current directory (so I end up with hardcoded POM’s basedir in the generated Ant file). Neither of suggested solutions to escape interpolation worked for me: \${basedir}, $${basedir}, ${dollar}{basedir}. Whatever I did – Maven (v2.2.0) was pretty determined to make a replacement! In the end, I’ve decided to trick it and generate a file in two phases:

1. I’m writing just {basedir} to the file and no Maven interpolation is triggered
2. I append a $ to it with \$$$1: $1 is original ${basedir} matched by <from> pattern and \$$ is a dollar .. yeah, it took me some time to figure it out)

                                

<resource>
    <targetPath>${directory.to.copy.the.file.to}</targetPath>
    <file>${directory.to.take.the.file.from}/build.xml</file>
    <replace>
        <to>
            <![CDATA[
<project default="run">
    <target name="run">
        <exec executable="{basedir}/some.exe" spawn="true"/>
    </target>
</project>
            ]]>
        </to>
    </replace>
</resource>
<resource>
    <targetPath>${directory.to.copy.the.file.to}</targetPath>
    <file>${directory.to.copy.the.file.to}/build.xml</file>
    <replace>
        <from>(\{basedir\})</from>
        <to>\$$$1</to> <!-- To get a "${basedir}" in output -->
    </replace>
</resource>

I now have a ${basedir} expression in my generated Ant file! Never imagined it could take so long .. But that what makes our profession so nice, isn’t it ?

Maven – Packing and unpacking archives in Mojo with TrueZip

Note: an update is available.

Normally, I don’t like to write about things … hmm … that I don’t like (like bad movies or bad libraries) – simply because there are so many of them! Talking about fun things (lie Groovy and Maven :)) is what makes it all look a little bit better than it really is ..

But somehow there’s this thing that really annoyed me recently – a TrueZip library. This should be a Java API for working with archives via virtual file system. Sounds big but all I needed was to pack or unpack a couple of ZIPs, nothing fancy. Unfortunately, I couldn’t. I kid you not – there are numerous de.schlichtherle.io.File methods but I didn’t manage to pack a ZIP file in a bearable amount of time (an hour or two). Call me stupid but files were copied from place to place, other things were happening, exceptions were thrown but a simple “zip” operation, you know – I didn’t get to it. archiveCopyFrom(), archiveCopyTo(), whatever … I had some success with packing a directory, however – surprisingly (yes, I was surprised when things were happening out of sudden the way I’ve expected), archiveCopyAllFrom() did the job. Same thing happened with “unzip” – I couldn’t unpack a ZIP archive either 😦

I think that’s a good example of overly complex and not intuitive API. I bet I could figure out how to pack or unpack a file after RTFM-ing for half a day but alas, I’m not that patient for such a basic thing. TrueZip Maven Plugin to the rescue! Thanks God someone have wrapped this library with a more meaningful API (although I still don’t get how one talks about ZIP-related library and never mentions words like “(un)zip” or “(un)pack” – only “copy”, like in “copy to/from archive”).

Anyway, using FileSet and DefaultTrueZip#copy() one can easily pack and unpack archives in Mojo:

import org.apache.maven.shared.model.fileset.FileSet;
import org.codehaus.mojo.truezip.util.DefaultTrueZip;

...

getLog().info( String.format( "Packing [%s] to [%s]", 
                              sourceDirectoryPath, 
                              destinationArchivePath ));

FileSet fs = new FileSet();
fs.setDirectory( sourceDirectoryPath );
fs.setIncludes( includes );
fs.setExcludes( excludes );
fs.setOutputDirectory( destinationArchivePath );

new DefaultTrueZip().copy( fs, false, getLog());
de.schlichtherle.io.File.umount(); // To update (flush) the file

getLog().info( String.format( "[%s] packed to [%s]", 
                              sourceDirectoryPath, 
                              destinationArchivePath ));

...

getLog().info( String.format( "Unpacking [%s] to [%s]", 
                              sourceArchivePath, 
                              destinationDirectoryPath ));

FileSet fs = new FileSet();
fs.setDirectory( sourceArchivePath );
fs.setOutputDirectory( destinationDirectoryPath );

new DefaultTrueZip().copy( fs, false, getLog());
de.schlichtherle.io.File.umount(); // To update (flush) the file

getLog().info( String.format( "[%s] unpacked to [%s]", 
                              sourceArchivePath, 
                              destinationDirectoryPath ));

...

<dependency>
    <groupId>de.schlichtherle.io</groupId>
    <artifactId>truezip</artifactId>
    <version>6.6</version>
</dependency>

<dependency>
    <groupId>org.apache.maven.shared</groupId>
    <artifactId>file-management</artifactId>
    <version>1.2.1</version>
    <scope>compile</scope>
</dependency>

<dependency>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>truezip-maven-plugin</artifactId>
    <version>1.0-beta-2</version>
    <scope>compile</scope>
</dependency>

That’s it.
I wish I could simply use TrueZip for that but, apparently, I’m not that smart to get it’s API. I just want to get my work done, you know …

Update: I’ve followed plugin’s code to see how DefaultTrueZip works but same API calls just didn’t work for me! I suppose it’s a mutual hatred from the first sight.

Groovy/Maven – Multiline Strings

Groovy multiline Strings are always using “\n” as line terminator and sometimes this may be a problem but a simple replace() solves it:

"""
<?xml version="1.0" encoding="UTF-8"?>

<project xmlns              = "http://maven.apache.org/POM/4.0.0"
         xmlns:xsi          = "http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation = "http://maven.apache.org/POM/4.0.0
                               http://maven.apache.org/maven-v4_0_0.xsd">
""".replace( "\n", "\r\n" )

I had this issue when ran the above multi-line replacement today (yeah, I’d like all POMs to have this nice header) – some files that already had this header were still modified by the script as it replaced “\r\n” that were previously in a file by “\n” – that’s when I had to add the replace() call.

Just something to keep in mind.

Update: same issue happens in Maven when it passes a multiline plugin’s configuration to Mojo (like replacement content between <to> tags) – it joins the lines with “\n” as well. If they’re to be written to a file – s.replace( "\n", System.getProperty( "line.separator" )) is required. Note that in Groovy version (above) I did not use System.getProperty() and that’s because we want all POMs have “\r\n” line-endings regardless of OS the script is running on (but we’re mostly using Windows anyway).

Maven – reusing other plugins is .. Hard

I suppose each new technology has it’s own serious flow, right after it is invented. The problem is not with the flaw (we all make mistakes and nobody can predict how things will be used) but with the fact that it’s not always easy to fix it later. Some decisions once made – may stay for a long time and that’s the problem.

Ant, for example, made an assumption there should be no logic and no state in builds: no if’s, no loops, no try-catch, no mutable variables. “Big mistake!” – remember ? 🙂 Of course, everybody had if’s and loops and I guess lot’s of people went crazy seeing “immutable properties” in action. Fortunately, there was Ant-Contrib to rescue everybody and allow to set variables, for God’s sake.

Now Maven – it has it’s own issues like not thread-safe local repository and no built-in support for parallel builds or downloads but those may be fixed seamlessly for all users, we won’t even notice. I guess it’ll be announced someday that local repository is finally thread safe and parallel builds and downloads are supported out-of-the-box.

But it has another flaw that bothers me much more – plugins re-usability. It just doesn’t exist, same way as no loop existed for Ant. I happen to write some Maven plugins those days and there’s no built-in way one Maven plugin can use another! May be it’s just the way it should be – after all, I suppose no Firefox extension can use another as well. But you see, Maven plugins are really simple objects with getters, setters and some Javadoc annotations (real Java 5 annotations are only available with AnnoMojo but Maven guys have also started to think in the same direction).

Maven plugin (Mojo) is a simple Java object – still, you can’t easily re-use it!

1. Why would you want to re-use it?

   Because there are so many plugins dealing with lot’s of Maven details you really don’t want to know about.
   Like copying an artifact to some folder, given it’s groupId, artifactId, and version (this artifact may be available locally [where?], remotely [where?], it needs to be downloaded, etc, etc). Yes, there’s a “maven-dependency-plugin” but I had to write a Mojo that is doing basically the same, still in a different way. CopyMojo is already doing what I need so how hard can it be to re-use this logic? And here we come to the second question ..

2. Why do you think you can’t easily re-use another plugin ?

   Because that’s Maven’s flaw I mentioned in the beginning – Maven has an assumption that only container will ever initialize a plugin. You can’t just new CopyMojo() – you’ll get a new object but no fields will be injected by the container. And if there are no setters for them – you can’t initialize them either! How about MyMojo extends CopyMojo ? (ignoring the fact that I had a convenient ParentMojo for all plugins that was abandoned in this case). Another catch here – Maven doesn’t inject super-class fields (AnnoMojo does that, but CopyMojo isn’t using it). All my Mojo annotated fields are properly initialized but all super-class fields are not, they’re null. I had to declare annotated duplicates of all super-class fields (carefully giving them another name to avoid confusion) and initialize them by myself:

   this.project     = getMavenProject();
   this.factory     = getArtifactFactory();
   this.resolver    = getArtifactResolver();
   this.local       = getArtifactRepository();
   this.remoteRepos = new ArrayList<ArtifactRepository>( getRemoteArtifactRepositories());

   All getters are mine, they retrieve values of duplicate fields and initialize those from a super-class.

   Btw, even if sub-classing had worked without duplicate fields – it is still an ugly way to re-use a plugin!

3. Why do you think it is a serious flaw ?

   Because today Maven provides no support for plugins repository or plugins factory or whatever you call it. Something that you could ask for an instance of plugin (providing it’s groupId and artifactId) that is fully initialized (= all fields injected) and ready to be configured.

4. Are there any solutions available ?

   Yes. Jade Plugins: “Using Annotated Plug-in Mojos from Maven Anno Mojo, jade plugins can inherit and delegate to each others, without copy/pasting parameters and code”.

   Seems exactly what I need so I’ll try it soon.

But I do hope there will be plugins factory one day!

Update: thinking about it again – the container itself knows how to load a plugin given it’s groupId and artifactId. So all I need is to find out how it’s implemented and re-use this API, if possible.

Maven – files included by FileSet

Btw, in case you was wondering (like I did one day) how one gets (in a Maven plugin) all files included by a FileSet, here’s the answer:

import org.apache.maven.shared.model.fileset.FileSet;
import org.apache.maven.shared.model.fileset.util.FileSetManager;
import java.util.Arrays;

...

FileSet fs = new FileSet();
fs.setDirectory( ".." );
fs.setIncludes( Arrays.asList( "..", ".." ));
fs.setExcludes( Arrays.asList( "..", ".." ));
String[] files = new FileSetManager().getIncludedFiles( fs );

Dependency – org.apache.maven.shared:file-management (javadoc, repo1, examples):

<dependency>
    <groupId>org.apache.maven.shared</groupId>
    <artifactId>file-management</artifactId>
    <version>1.2.1</version>
    <scope>compile</scope>
</dependency>

Maven – folders excluded by default

While working on a Maven plugin that copies resources (similarly to “maven-resources-plugin” but with a few important differences, more on that later) I had to put a list of folders excluded by default.

Finally found it here: org.codehaus.plexus.util.DirectoryScanner.DEFAULTEXCLUDES.
It is part of org.codehaus.plexus:plexus-utils (javadoc) that is brought by org.apache.maven:maven-project (javadoc) which is used by plugin authors for many things (by having MavenProject as Mojo parameter), one of them is accessing Maven properties:

@MojoParameter( expression = "${project}" )
private MavenProject mavenProject;
public  MavenProject getProject ()
{
    return this.mavenProject;
}

…

getProject().getProperties().setProperty( propertyName, value );

So, DirectoryScanner.DEFAULTEXCLUDES:

• **/*~
• **/#*#
• **/.#*
• **/%*%
• **/._*
• **/CVS
• **/CVS/**
• **/.cvsignore
• **/SCCS
• **/SCCS/**
• **/vssver.scc
• **/.svn
• **/.svn/**
• **/.arch-ids
• **/.arch-ids/**
• **/.bzr
• **/.bzr/**
• **/.MySCMServerInfo
• **/.DS_Store

A bigger list is available through hidden.org.codehaus.plexus.util.AbstractScanner.DEFAULTEXCLUDES inside “apache-maven-2.2.0/lib/maven-2.2.0-uber.jar” but I don’t see how it can be used by Maven plugin (besides copy-pasting the list manually, of course):

• **/*~
• **/#*#
• **/.#*
• **/%*%
• **/._*
• **/CVS
• **/CVS/**
• **/.cvsignore
• **/RCS
• **/RCS/**
• **/SCCS
• **/SCCS/**
• **/vssver.scc
• **/.svn
• **/.svn/**
• **/.arch-ids
• **/.arch-ids/**
• **/.bzr
• **/.bzr/**
• **/.MySCMServerInfo
• **/.DS_Store
• **/.metadata
• **/.metadata/**
• **/.hg
• **/.hg/**
• **/.git
• **/.git/**
• **/BitKeeper
• **/BitKeeper/**
• **/ChangeSet
• **/ChangeSet/**
• **/_darcs
• **/_darcs/**
• **/.darcsrepo
• **/.darcsrepo/**
• **/-darcs-backup*
• **/.darcs-temp-mail

Enjoy ! 🙂

Update: the situation described above is correct only for org.codehaus.plexus:plexus-utils:1.5.6, in 1.5.15 – DirectoryScanner has a bigger list similar to that in “maven-2.2.0-uber.jar”.

I’m adding explicit plexus-utils version to my plugins POM:

<dependencies>
    <dependency>
        <groupId>org.apache.maven</groupId>
        <artifactId>maven-plugin-api</artifactId>
        <version>2.2.0</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.maven</groupId>
        <artifactId>maven-project</artifactId>
        <version>2.2.0</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.codehaus.plexus</groupId>
        <artifactId>plexus-utils</artifactId>
        <version>1.5.15</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.maven.shared</groupId>
        <artifactId>file-management</artifactId>
        <version>1.2.1</version>
        <scope>compile</scope>
    </dependency>

    <!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
    <!-- http://www.jfrog.org/sites/mvn-anno-mojo/latest/ -->
    <!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->

    <dependency>
        <groupId>org.jfrog.maven.annomojo</groupId>
        <artifactId>maven-plugin-anno</artifactId>
        <version>1.3.2</version>
    </dependency>
</dependencies>

Maven – inheritance

One of Maven features not appreciated enough (from my experience) is inheritance. Many people are used to the fact that each module had a “build.xml” once. Those Ant files were usually big and they were 80% (or more) copy-pasted since Ant’s support for inheritance was terrible (<import> was supposed to do the job but I never saw anyone using it for actual inheritance – I myself only used it for loading <macrodef> libraries). So it was Ok to have those big Ant files, one per-module. Almost nobody knew for sure what they were doing and almost nobody was capable (or willing) to maintain them – well, that what happens when you copy-paste your build logic and Ant tasks all over the place. But I don’t blame anyone – there was almost no choice with Ant ! (sometimes I hate it as much as I like it: after all, crafting a cool <macrodef> may be a real fun .. but since I’m in a Groovy camp now – I guess those <macrodef> times are also gone for me)

Anyhow, we’re with Maven today and it supports inheritance. This time it’s for real. Any plugin configured in <parent>’s POM will be invoked automatically (unless <inherited> is false). This can be pretty powerful! The only problem left is people not looking for a ways to make their POMs smaller by inheriting <parent> behavior (plugins). I saw developers still copy-pasting plugin configurations from POM to POM, just out of habit doing the same with Ant, I suppose. Not surprisingly, they end up with exactly the same situation they had before – a biiig POM, one per-module.

We can do better! How about this one:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns              = "http://maven.apache.org/POM/4.0.0"
         xmlns:xsi          = "http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation = "http://maven.apache.org/POM/4.0.0
                               http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>com.company</groupId>
    <artifactId>VisualStudioModule</artifactId>
    <packaging>jar</packaging>
    <name>[com.company:VisualStudioModule]-[SVN/Path]</name>


    <parent>
        <groupId>com.company.visualstudio</groupId>
        <artifactId>main.2005</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>
</project>

This module is a VisualStudio project and parent POM (com.company.visualstudio:main.2005) invokes Ant which, in turn, invokes “devenv.exe” of VisualStudio 2005. There’s a com.company.visualstudio:main.2008 <parent> as well, doing the same for Visual Studio 2008.

The point is – all I had to do in module’s POM is specify a correct <parent>. No copy-paste and POM itself can’t be any smaller. Having lot’s of similar modules – one would really appreciate carrying around those “smart parents” (as I call them) knowing what to do, once they’re referenced.

We also use this technique for packaging similar archives (there are many modules, each one producing a ZIP with a predefined structure) – we have a parent POM knowing exactly how to pack an archive and all other POMs just reference it, adding a little bit of their own configurations (by further configuring parent’s plugins but I usually prefer defining simple variables that impact the way parent plugins are running – usually, they’re passed as arguments). That’s template pattern in action where parent knows 90% or even 100% of what needs to be done so children have very few work left to do.

This kind of inheritance is a very powerful thing I see in Maven so I suggest to use it more.
I guess nobody wants to end up with tons of huge and almost-similar POMs, spread all over the source tree … Those ancient days are long gone already!

Maven – replacing < name > with something more meaningful (groupId:artifactId)

Maven uses POM’s <name> tag content when printing:

• Reactor build order
• Building a module
• Reactor Summary

From my experience, people don’t really pay much attention to that tag. Some just copy <artifactId> there (that was me for a long time), others provide kind of project description there.
But whatever the case is – it usually takes some time to understand what Maven is building right now when there are lot’s of modules running since all that’s displayed is <name>! Personally, I never understood why someone would print this <name> tag instead of “<groupId>:<artifactId>” (even opened a JIRA case) – after all, I think their combination provides a pretty good description of a project (in fact, when <name> is missing – that’s exactly what Maven prints).

But now, after having a script (which is very different today from that initial version, my Groovy skills are improving) to iterate over all POMs in order to replace their <groupId> – it was a natural step for me to replace <name> tags as well: 🙂

text.replaceFirst( /<name>(.+?)<\/name>/,
                   "<name>[${ groupId }:${ artifactId }]-[${ svnPath }]</name>" );

Where ${groupId} and ${artifactId} are taken from the POM and ${svnPath} is POM’s path in SVN.

Something like:

<name>[com.company:PomChanger]-[Some/Folder/projects/PomChanger]</name>

Now, when Maven runs we see bunch of new <name>s displayed giving an exact and immediate information about what POMs are aggregated and built.

And it looks lovely in Hudson too!

It now runs automatically (using GMaven plugin) as part of Hudson nightly build so each new POM gets a correct <name> on the following day.