Out of SourceForge, all in Bintray

Now that Objenesis is in Bintray. The goal was to also have EasyMock binaries at the same place. They were hosted on SourceForge. And as you might have read, SourceForge is now on the dark side so going out of there was becoming a priority.

This time it was easier. I basically made a list of versions and release dates.

And then I made a loop around the lines to the

  • wget to get the binary and store it keeping the SourceForge structure
  • curl to create the version in Bintray
  • curl to upload the binary
I've manually added the release notes in GitHub but wasn't able to figure out how to push them to Bintray. If someone knows, please tell me.

To be helpful, here is the loop that I used to create the versions.


# This is because I want the file upside down to create the older versions first
tail -r $1 > tmp.tsv

while read p; do
version=$(echo "$p" | cut -d' ' -f1)
date=$(echo "$p" | cut -d' ' -f2)

version=${version// /-}

echo $version $date

content="{ \"name\": \"$version\", \"desc\": \"$version\", \"released\": \"${date}T00:00:00.000Z\", \"github_use_tag_release_notes\": false, \"vcs_tag\": \"$version\" }"
echo "$content"

curl -v -XPOST -H "Content-Type: application/json" -H "X-GPG-PASSPHRASE: ${PASSPHRASE}" -uhenri-tremblay:${API_KEY} \
-d "$content" \

done <tmp.tsv

Migrating EasyMock Jira to GitHub

Following my previous article, I will now describe the Jira migration of EasyMock to GitHub issue tracker.

EasyMock Jira was on Codehaus. So I had to migrate it somewhere else. Instead of migrating to another Jira instance I've decided that EasyMock doesn't need all that complexity anymore and that going to Github would enough.

The migration was made in two steps:

  1. Save everything on my disk
  2. Import everything in GitHub
The first step is important to keep all the original content in case it is needed.

The final result is nice but some things can't be done:
  • Original authors of issues and comments are lost. They are now mentioned in the description but they are not the real authors anymore
  • Creation dates, modification dates, etc are lost
The final code is here. If you need to do something similar it will be useful, however it can't be reused as is since it's not that clean. I also did some parts manually as you will see. 

Export (Export.java)

I used the Spring RestTemplate to retrieve my issues through the Jira API
  • The EASYMOCK project
  • All issues in the project 
  • All fields to keep the meaning of the custom fields
  • All priorities to have the list
  • The project components
  • The project versions
  • The description of all users that have created or fixed an issue
Then I've downloaded all attachments with wget. I've create the list of attachments using a basic shell command (grep attachme issue*.json | grep content | cut -d: -f4). Not pretty but it works.

Everything is on my disk. Now I'm safe.

Import (Import.java)

The import was made in many steps. All could easily automated but it was quicker to just do them manually.

GitHub doesn't have all Jira functionalities. There are no resolution field, affected version,  priority, attachments or custom fields. So things need to be adapted.
  • Affected version is dropped
  • A fixed version becomes a milestone
  • A component is a label
  • A resolution is a label
  • Custom fields are dropped
  • A priority is a label
  • Attachments a pushed in a dedicated repository


All labels were created manually. However you can look at CreateLabels.java to see how it could be automated.


Each Jira version became a milestone. This was done automatically using the Github API. The code is in CreateMilestones.java.


GitHub only supports images in attachment. The classical way to do an attachment of something else is to do a gist. I had a look at that and that's why you will see some code to create a gist in GitHubConnector.java.

However, I took another because I had TARs, ZIPs, JARs and other binaries in my attachment. So I've decided to push everything to a dedicated github repository and then reference the files in my issues.


The final step was to migrate the issues themselves. I had a quick look at the code used to migrate from Google Code to GitHub. There are two importants tricks in there.
  • You can't insert issues too quickly because you can go over the GitHub rate limit
  • You need to put some time between two comments to keep them in order because GitHub sorts the comments according to the timestamps
As soon as you know that, it's basically a mapping from fields to fields. The only final trick is that GitHub issues only have two status: Open and Closed. A newly created issue is always in the Open status. So you then need to modify it to close it if needed.

What is left to do

I'm pretty much done for now. The only thing that is still missing it the release notes. Jira provides a page giving the release notes for each version. I need to regenerate them from the issues on a milestone.

But I'll try to do some actual EasyMock code first... Cheers!

Virtual goods are still mine: Open letter to The Economist

I'm inaugurating a new section that will be more life oriented i.e not talking about software engineering. It might be in English or French depending on my mood and the public expected to read the entry. So this first blog post is more or less an open letter to The Economist.

When I'm not doing software engineering, I'm highly interested in finance, economy and politics. Until recently, I was even a diligent subscriber of The Economist. I've decided two months ago to not renew my subscription. Not because I don't like the magazine anymore but because it was too time consuming to read. I want this year to have the time to read books.

I was receiving the paper version which allows me to have access to the online version. Of course, I'm not keeping the paper versions in my basement since I was having access to them online. I was really surprised to realize that as soon as my subscription expired, I've lost the access to the entire The Economist website content. Event the editions THAT I PAID FOR!

I found that to be quite a cheap decision from a journal as renowned as The Economist.

But I also think it shouldn't be legal. It's been one of my thoughts for a while now.

Virtual properties should have the same "rights" as physical ones. When I buy a physical book, I can sale it, lend it, give it.

If an buy a e-book, I can't do all this. I'm locked in whatever rules the seller wants to apply. And I don't feel good about it. I'm pretty sure I'm not the only one and I'm also pretty sure it creates adoption issues.

Of course you can buy virtual gears. Sellers love to sell you stuff. Virtual or not. But when it's virtual, they currently make sure you will never really be the owner of it.

Yes, I do understand the concept of service. For instance, The Economist offers the website as a service to their customer. They pay the hosting, the development so it makes sense that if I stop paying, the service stops. But it doesn't make sense that I can't download the PDF of all the issues I bought. Of course it means that I would then be allowed to illegally diffuse them. But I can allow photocopy my paper issue. Or scan it.

Meanwhile, I'm left with a bitter taste in my mouth. Until these's a law protecting my virtual properties, I'm not really sure I will trust The Economist anymore.

Finish migrating Objenesis to GitHub

With Codehaus and Google Code closing, I’m happily required to migrate Objenesis and EasyMock.

Both projects have an hybrid hosting using different platforms. I’ve decided to start with Objenesis which is easier to migrate. I’ll describe the process in this post. Hopefully, it will be helpful to someone but, in fact, I’m also looking forward to your feedback to see if I can improve my final setup.

So, Objenesis source code was originally on Google Code. I’ve moved it to GitHub some years ago mainly because pull requests are awesome. The website is also a GitHub page. Part of the documentation is on Google Code wiki, the binaries are on Google Code as well as the issue tracker.

Google being nice folks, they used to provide a nice way to migrate everything to GitHub. But I couldn’t use that because

  • I don’t want the project to end up in henri-tremblay/objenesis. I want it where it is now, in the EasyMock organisation
  • I only want to migrate the wiki and the issues since the sources are already there

So here’s what I did instead.

Issue tracker

The issue tracker was migrated using the IssueExporterTool. It worked perfectly (but is a bit slow as advertised).

Wiki pages

At first, I tried to export the entire Objenesis project to Github to be able to retrieve the wiki pages and then move them to the real source code. The result was quite bad because the tables are not rendered correctly. So I ended up manually migrating each page to markdown. There was only 3 pages so it wasn’t too bad.


This was more complicated. Maven binaries are already deployed through the Sonatype Nexus. But I needed to migrate the standalone bundles. I’ve looked into three options:

  • GitHub releases
  • Bintray
  • Both (GitHub releases exported to Bintray)

GitHub releases are created automatically when you tag in git. But you also seem to be able to add some release notes and binaries over that. I didn’t want to dig too much into it. I knew I wanted to be in Bintray in the end. And I wanted easy automation. Bintray was easy to use so I went for Bintray only. Be aware, the way I did the migration is low-tech (but works).

  1. Make a list of all the binaries to migrate
    • Get them with wget https://objenesis.googlecode.com/files/objenesis-xxx.zip
  2. Create an organisation, a distribution repository and an [objenesis package])(https://bintray.com/easymock/distributions/objenesis) in Bintray
  3. Add my GPG key to my Bintray account
  4. Upload everything using REST (curl -T objenesis-xxx.zip -H "X-GPG-PASSPHRASE: ${PASSPHRASE}" -uhenri-tremblay:${API_KEY} https://api.bintray.com/content/easymock/distributions/objenesis/xxx/objenesis-xxx-bin.zip?publish=1)

It works, the only drawback is that no release notes are provided. They’ve always been on the website

Project moved

Finally, I’ve set the “Project Moved” flag to target GitHub. This quite aggressively redirects you to GitHub if you try to access https://code.google.com/p/objenesis.

What I learned today

When creating this blog, the first thing I wanted to talk about are those things you learn everyday when fighting on a technical issue. Because if it happened to me, it will probably happen to you.

What I discover is that a lot of these things are too small to be a blog post or will in fact just repeat a really nice existing blog post where I found my answer in the first place. And a blog post just telling you to read another blog post is not that useful.

Instead, I've decided to tweet these blog posts each time I fell on one. I'll use the #WhatILearnedToday hashtag.  If you're interested, you can then follow me on Twitter.

See ya!

Microbenchmarking is fun

But time consuming. So always remember that I should have, for instance, read the assembly generated code to be able to explain why an implementation is faster than another. But I don't have that kind of time right now. So I was just toying around.

Anyway, a friend of mine has replaced a bunch of String.replaceAll with a way longer but way faster implementation based on a for loop. I was happy about it.

To make a long story short, it has bounced a bit around the internet and has created an official challenge on github. All this is well resumed (in French) by Olivier Croisier on his blog.

Since I have an interest in performance tuning, I've started to play with it. Being annoying, instead of providing my own implementation, I first had a look at the benchmark implementation itself. Done with JMH (always reassuring to see a benchmark that isn't home made).

So, first I asked for unit tests for make sure my implementations were accurate. Then I've extracted the benched methods in a specific class. At first, it was a static class inside the JMH benchmark. JMH is rewriting this class into something else. To be safe, I prefer to put the benched code where it will be for real.

After that, I notice an issue. The dataset used was randomly generated at the beginning of the benchmark. So each benched method was receiving a different dataset. Not good to make comparable results. We are now generating a dataset first and then run all benchmarks on it.

Finally, I wanted the dataset to be representative of a real file. Some I made some statistics on a real file to get the usual length of lines, occurence of line feeds, etc. This allowed me to change the generator to create a better result.

Now I was set. I must confess, my best result is a plain copy of Cédric Champeau's solution with a tiny tuning. I made sub-methods. On my machine (OS X Yosemite, 2.6 GHz Intel Core i7, Java HotSpot 64 bits 1.8.0_40), it's 10% faster. On Travis, it's not that conclusive.

Anyway, now the funny part is that I tried lots of things that should have been more efficient but that are not.

  • Putting the pattern of the regex in a constant should be faster than only recreating it all the time. No. Pretty much the same result
  • Adding the missing else in two subsequent ifs should prevent a comparison and be faster. No. It's almost slower
  • Using the ternary operator instead of a if can be a bit faster. But not much. But it is different.
  • Removing a ++ that was redundant doesn't change anything
  • Use a new array instead of cloning and then writing over a single array. This removes a huge copy. But it's slower. Probably because of page faults.
  • Using arraycopy instead of assigning every single character. Much slower
  • Using Unsafe to retrieve, allocate and assign. Also much slower. But I'm pretty sure I can improve this one
As I said, I've not tried to explain the results. I'm just assuming they are valid. Please don't do this at home.

However, if the results are indeed valid, it seems that the JVM optimisations give counter-intuitive results. Sometime, doing more things if in fact faster.

And of course, if you want to try the challenge, it's open to all :-)

The history of partial mocking

Someone asked me this week to give some legitimate reasons to use partial mocking. It's indeed a good question. Especially because the EasyMock documentation is explicit on telling that using it is probably a code smell.

The funny thing about partial mocking is that I've invented it to workaround issues we had at that time (2004... a long long time ago in the computer engineering world). It's only later that we've discovered some legitimate usages.

You see, when I first started to work on class mocking, it wasn't possible to bypass the constructor yet (or at least, I wasn't yet digging in the JVM far enough to be able to do so). So, I was using a lot of deductions to find which constructor will be the best to use. The algorithm was something like this:
  1. Try to use the default constructor
  2. If it's not there, use the constructor with the less parameters and create a mock for each of these parameters
  3. Pray it won't fail when creating the new instance
Frequently, this method would fail because some nasty individual had put really agressive code in a constructor. For instance, I've seen constructors opening sockets...

So, to workaround that, I made it possible to select the constructor to use and also to pass real arguments to it. The arguments were not used afterwards. They were just there to prevent the constructor from crashing.

But then, something worse happened. Some constructors were calling other methods... And since no expectations were set (of course! The mock doesn't exist yet!), mock creation was failing.

Partial mocking was born to solve that.

I made it possible to prevent some methods to be mocked so when the constructor would call them, they would behave as usual and I'll get my mock. TADA!

Yes, it was ugly, hacky and I wasn't really proud of it but that was the best I could do at that time (remember: 2004).

Not so much later, I found a way to bypass the constructor entirely the same way HotSpot was doing it during serialization. The main drawback was that EasyMock was now working only on HotSpot since the code was HotSpot specific.

The good news are that this code was already used by a bunch of other frameworks like XStream for instance. So other JVMs were starting to be compliant. I remember asking the JRockit team about it and two minor versions later, it was there.

Still, that's was caused the creation of Objenesis. The magic library that creates objects without calling the constructor on any JVM. In fact, Objenesis is not that useful anymore because we have Unsafe.allocateInstance. Unsafe also is OpenJDK specific but so many frameworks are using it that pretty much everyone has implemented it. However, I'm still using Objenesis for two reasons:
  1. You never know when a JVM won't be compatible with Unsafe
  2. My benchmarks have shown the using JVM specific code is way faster than using Unsafe
So I'm staying on Objenesis for now (and there is an instantiator using Unsafe so there's no drawback using it).

Anyway, back to partial mocking.

So, we are now able to bypass the constructor and partial mocking was created because we were not able to. Why is it still there?

There are two main usages. The first and most legitimate one is to test the Template Method Pattern. Let's say you have an abstract base class with abstract methods and concrete methods.
public abstract class BaseClass {
public boolean beTruthy() {
// stuff
boolean b = doSayTheTruth();
// other stuff
return b;

protected abstract boolean doSayTheTruth();
You want to test the concrete ones.
  • You could create a fake implementation, but that's annoying. 
  • You could just use a real implementation but then it will make you test more than needed. 
  • Or you could create a partial mock. 
In fact, when creating a partial mock, EasyMock now automatically considers that abstract methods will be mocked and concrete methods won't.

So you could do
BaseClass myClass = createMockBuilder(BaseClass.class).createMock();
How sweet.

The other reason is a little bit less legitimate. Let's say you have a class with a bunch of methods calling each others.

You would like to test the class but testing everything at once is nearly impossible.

A good solution is to test the methods one after the other by using partial mocking.
  1. You will test the first method by mocking everything it calls. 
  2. You will then test another method down the stack by mocking everything it calls. 
  3. And so on and so on.
Of course, that's bad code! But if there are no tests, you can't refactor! Step one, the tests. Step two, refactor. And yes, during the refactoring, the need for partial mocking should disappear. But that doesn't mean partial mocking isn't helpful.

That's all for today. Happy partial mocking. I hope you find this story interesting and/or useful.

Stubbing with fluent-http

One thing I do quite often if to make sure I can lunch my application offline. For instance, my application needs to access an external system that is only available from my company network.

So if I'm offline, it just doesn't work. And I don't like my ideas to go on hold just because I'm not in front of my desk at work or because the remote service is down.

Normally I would launch an embedded something (usually jetty) and have it server my stuff. The trick is to save real results from the actual server in some files and then serve these files. It's a bit complex to configure but in the end it works.

I use Spring profiles to turn the fake mode on and off.

This time, I tried something different. I used fluent-http. It is a really lightweight web server that gets configured a bit like NodeJS would.

I'm quite happy about it so I thought I should share.

My first usage was to turn a kinda integration test into a unit test. So I have a test that used to call the real server and that is now calling my fake server. I can in fact launch it in both modes. That's useful to make sure my code it still compliant with the actual server implementation.

It looks like this:

@Beforepublic void setup() throws Exception {
if(fake) {

private void initFakeServer() {
server = new WebServer().configure(routes -> routes
.get("/api/stuff", requestContent("stuff.json")))

@Afterpublic void tearDown() {
 if(fake) {

private Payload requestContent(String file) {
try {
String content = new String(Files.readAllBytes(Paths.get("src/test/data/stuff", file)), "UTF-8");
return new Payload("application/json;charset=UTF-8", content);
} catch (IOException e) {
throw new RuntimeException(e);
Nice and sweet. Fluent-http can also serve static files directly based on the extension if you prefer. But you will then need to match your directory layout with was is normally served by the real server.

Two gotchas though:

  • The content type can normally be deduced from the filename extension. However, the json extension wasn't supported. I made a pull request that was merged today so it should be fixed soon
  • The dependencies in the maven pom file are a bit more fast than what I was expecting. It might clash with some of your. I'm currently excluding slf4j-simple. This one should probably be in scope provided in the code

Syntax highlighting in PowerPoint

Today I was wanting to copy some code in a PowerPoint presentation (yes, I still think PowerPoint is the best presentation tool there is).

On Windows, it's pretty easy. I used to use Notepad++ to copy the text including the highlighting and job done.

On a Mac, it seems a bit more complicated. There seem to be some IntelliJ plugins but I had no luck with them.

I ended up using a neat trick. It's not great but at least it works. One of the main drawback is that it always copy the entire file. And Powerpoint seems to have a bug. When you copy something that is larger than the slide, you don't get to have the "Keep source formatting" option.

Anyway, it's cumbersome but it works.

First you need pygments. Which is installed using pip
sudo easy_install pip
sudo pip install pygments

Then you can do that:

pygmentize -f rtf Allo.java | pbcopy

It will highlight the file and put it in the clipboard. Tadam!

But I'm still open to any answer that is simpler than that...

UPDATE: It seems that the IntelliJ plugin 'Copy' on steroid is now working on my machine. It just magically copy as RTF text. Really useful 

Extract release notes from Jira

If you are like me, you love automation. But sometimes, software and automation just don't like each other.

Today, I wanted to do a really simple thing. I wanted to retrieve the release notes for an EasyMock version on Jira. Seems easy enough. Jira has a great REST API that should allow to do that in two seconds.


Let's say you want the release notes for version 3.2. There is no REST API for the release notes. There is one for the versions. However, to get the details of a version, you need the version id. Which isn't 3.2 but an internal id. And there is no search available on the versions.

I've decided to keep things "simple" by not using any high level language. Only bash. I'm pretty sure Gradle could have helped me but I didn't want to put my arm in there yet.

So, first step, get the id by grepping it from the list of all versions.

# Seems complicated but I'm just adding a backslash before the . in 3.2
escaped_version=$(echo $version | sed "s/\([0-9]*\)\.\([0-9]*\)/\1\\\.\2/")

# Retrieve all versions and the id. Hoping a bit that the fields will stay in the same order
jira_version_id=$(curl --silent "http://jira.codehaus.org/rest/api/2/project/EASYMOCK/versions" | grep -o "\"id\":\"[0-9]*\",\"description\":\"EasyMock $escaped_version\"" | cut -d'"' -f4)
Good. Now I have the version id. But sadly, the API doesn't give the release notes for this version. But the Jira GUI does. When doing it by hand, you click on the version, the on the release notes, the you select the format wanted (text or html) and finally, you reach a page with a text area showing exactly what I want.

But, of course, there's no way to get only the content of the text area. So I now need to extract it from the page

# Get the page
release_notes=$(curl --silent "$release_notes_page")

# Extract the interesting page
echo "$release_notes" | sed -n "/<textarea rows=\"40\" cols=\"120\">/,/<\/textarea>/p" | grep -v "textarea

Et voilà!

BTW, there is a Jira opened in Atlassian bug tracker asking for this exact feature. Please vote.