Future behaviour through time

Yesterday, I was playing with interruption. At some point I ended up with this code.

public <T> T uninterruptibleGet(Future<T> future) throws ExecutionException {
  while(true) {
    try {
      return future.get();
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
    }
  }
}

This is bad code. Don’t do that. It should be an infinite loop. For those not playing with interruption daily here’s why:

  1. Waiting on get
  2. Be interrupted
  3. Catch the exception and set the interruption state back (Thread.currentThread().interrupt())
  4. Go back to get
  5. Notice we are interrupted (the interruption state is on)
  6. Throw an InterruptedException
  7. Go to 4

The thing is that when running the code, there was no infinite loop.

So I wanted to know why.

The answer is in FutureTask.get().

public V get() throws InterruptedException, ExecutionException {
    int s = state;
    if (s <= COMPLETING) // here
        s = awaitDone(false, 0L);
    return report(s);
}

The get method first look at the state, if done, it just returns. If not, then it interrupts. Here we go, job done.

Then it goes to the Continuous Integration. And goes in an infinite loop. Whaaat???

At first I thought it was some race condition that was happening with other tests (they are run in parallel).

But no. The answer is much simpler than that: JDK 6

The FutureTask code above is from JDK 8. The JDK 6 code is different. It first starts to check the interrupt. So infinite loop it is.

Now, was is the right idiom for an uninterruptibleGet?

To get the long answer, I highly suggest that you go read Java Concurrency in Practice (chapter 7) right now. It is a must read for any Java programmer.

The short answer is this:

public <T> T uninterruptibleGet(Future<T> future) throws ExecutionException {
  boolean interrupted = false;
  try {
    while (true) {
      try {
        return future.get();
      } catch (InterruptedException e) {
        interrupted = true;
      }
    }
  } finally {
    if (interrupted)
      Thread.currentThread().interrupt();
  }
}

EasyMock generic typing

These days, I’m a bit annoyed about Java generics. Because it seems that when you want to be clean, you pretty much always get in troubles.

I’ll give you an example. The method to create a mock with EasyMock is currently typed like this:

public static <T> T mock(Class<T> toMock)

This seems quite straightforward.

But it is also quite annoying to use. For example, let’s try to mock a generic type:

List<String> list = mock(List.class);

It looks like something we would like to easily do but you get this nice warning:

Warning:(51, 33) java: unchecked conversion
  required: java.util.List<java.lang.String>
  found:    java.util.List

And there’s no way to get around it. Here are some attempts:

List<String> list = mock(List<String>.class); // no allowed (won't compile)
List<String> list = EasyMock.<String>mock(List.class); // won't compile either because the parameter type doesn't match)
List<String> list = (List<String>) mock(List.class); // still get a warning

So what’s the way out?

To be less accurate.

Yes. I can’t see any other way out. (Do you?)

So my plan is to change EasyMock typing with this:

public static <T> T mock(Class<?> toMock)

No relation anymore between the parameter and the returned type. WAT!!! Are you crazy?!?

So yes, this will compile without complaints :-( (but I can check the type coherence at runtime)

String list = mock(Integer.class);

But this will now work without any warning :-)

List<String> list = mock(List.class);

You can’t have your cake and eat it it seems. But as usual, I will be happy to be proved wrong.

Gatling Java DSL

My favorite stress tool has been Gatling for many years now.

Scenarios are

  • Easy to construct, It’s just code
  • Easy to version. It’s just code
  • Easy to run. It’s a simple command line that you launch in remote if needed
  • Having great result graphs
  • Easy to datamine. There is a simulation.log file that let’s you do some more digging when needed

My only complaints are

  • You can’t change the running scenario
  • It’s in Scala

The first one might be addressed by Gatling Frontline. I haven’t tried.

The second one is a bit more complicated to solve :-)

In fact, I don’t care that’s in Scala. But I would love to be able to core my scenario in Java.

So just for fun, I did a thought experiment. I’ve coded the Gatling API in Java and then coded the BasicScenario example using this API.

It’s not working for real

The API is just an emply shell. To know how different it would be from the Scala DSL.

The answer is: Not much.

My expectations were that I would need to rely on plenty of Java 8 features to get something close to the Scala DSl. But in fact, not even.

The main differences are:

  • Variables are typed
  • I’ve added a build method. But it’s just for style. I can get rid of it
  • A bunch of semi-colons
  • No syntactic sugar to create a map
  • I’m using Duration is used to time units
  • I need a run method where the setUp method is

That’s quite joyful and probably can be improved.

I’m now wondering if I could have used the Scala classes directly. I’m not good at making Java and Scala interacting.

So if someone is and want to try, please do so.

Manage emails in git

If you are like me, you tend to work on open source using your personal email and at YP using your enterprise email. But git doesn’t handle that well so you need to never forget to do

git config user.email henri.tremblay@somewhere.com

after each git init or git clone.

I finally had the time for work on an almost satisfying solution.

EasyMock 3.4 is out!

This version has finally removed the old (and ugly) partial mocking methods. You are now required to use the partialMockBuilder.

In exchange, you get really nice short methods to create your mocks: mock, niceMock and strictMock.

You also get a better stability since cglib and ASM are now embedded to remove a possible version mismatch with your own dependencies. Note that Objenesis will stay as an explicit dependency.

Change log

Out of SourceForge, all in Bintray

Now that Objenesis is in Bintray. The goal was to also have EasyMock binaries at the same place. They were hosted on SourceForge. And as you might have read, SourceForge is now on the dark side so going out of there was becoming a priority.

This time it was easier. I basically made a list of versions and release dates.

And then I made a loop around the lines to the

  • wget to get the binary and store it keeping the SourceForge structure
  • curl to create the version in Bintray
  • curl to upload the binary
I've manually added the release notes in GitHub but wasn't able to figure out how to push them to Bintray. If someone knows, please tell me.

To be helpful, here is the loop that I used to create the versions.

API_KEY=xxx
PASSPHRASE=xxx

# This is because I want the file upside down to create the older versions first
tail -r $1 > tmp.tsv

while read p; do
version=$(echo "$p" | cut -d' ' -f1)
date=$(echo "$p" | cut -d' ' -f2)

version=${version// /-}

echo $version $date

content="{ \"name\": \"$version\", \"desc\": \"$version\", \"released\": \"${date}T00:00:00.000Z\", \"github_use_tag_release_notes\": false, \"vcs_tag\": \"$version\" }"
echo "$content"

curl -v -XPOST -H "Content-Type: application/json" -H "X-GPG-PASSPHRASE: ${PASSPHRASE}" -uhenri-tremblay:${API_KEY} \
-d "$content" \
https://api.bintray.com/packages/easymock/distributions/easymock/versions

done <tmp.tsv

Migrating EasyMock Jira to GitHub

Following my previous article, I will now describe the Jira migration of EasyMock to GitHub issue tracker.

EasyMock Jira was on Codehaus. So I had to migrate it somewhere else. Instead of migrating to another Jira instance I've decided that EasyMock doesn't need all that complexity anymore and that going to Github would enough.

The migration was made in two steps:

  1. Save everything on my disk
  2. Import everything in GitHub
The first step is important to keep all the original content in case it is needed.

The final result is nice but some things can't be done:
  • Original authors of issues and comments are lost. They are now mentioned in the description but they are not the real authors anymore
  • Creation dates, modification dates, etc are lost
The final code is here. If you need to do something similar it will be useful, however it can't be reused as is since it's not that clean. I also did some parts manually as you will see. 

Export (Export.java)

I used the Spring RestTemplate to retrieve my issues through the Jira API
  • The EASYMOCK project
  • All issues in the project 
  • All fields to keep the meaning of the custom fields
  • All priorities to have the list
  • The project components
  • The project versions
  • The description of all users that have created or fixed an issue
Then I've downloaded all attachments with wget. I've create the list of attachments using a basic shell command (grep attachme issue*.json | grep content | cut -d: -f4). Not pretty but it works.

Everything is on my disk. Now I'm safe.

Import (Import.java)

The import was made in many steps. All could easily automated but it was quicker to just do them manually.

GitHub doesn't have all Jira functionalities. There are no resolution field, affected version,  priority, attachments or custom fields. So things need to be adapted.
  • Affected version is dropped
  • A fixed version becomes a milestone
  • A component is a label
  • A resolution is a label
  • Custom fields are dropped
  • A priority is a label
  • Attachments a pushed in a dedicated repository

Labels

All labels were created manually. However you can look at CreateLabels.java to see how it could be automated.

Milestones

Each Jira version became a milestone. This was done automatically using the Github API. The code is in CreateMilestones.java.

Attachments

GitHub only supports images in attachment. The classical way to do an attachment of something else is to do a gist. I had a look at that and that's why you will see some code to create a gist in GitHubConnector.java.

However, I took another because I had TARs, ZIPs, JARs and other binaries in my attachment. So I've decided to push everything to a dedicated github repository and then reference the files in my issues.

Issues

The final step was to migrate the issues themselves. I had a quick look at the code used to migrate from Google Code to GitHub. There are two importants tricks in there.
  • You can't insert issues too quickly because you can go over the GitHub rate limit
  • You need to put some time between two comments to keep them in order because GitHub sorts the comments according to the timestamps
As soon as you know that, it's basically a mapping from fields to fields. The only final trick is that GitHub issues only have two status: Open and Closed. A newly created issue is always in the Open status. So you then need to modify it to close it if needed.

What is left to do

I'm pretty much done for now. The only thing that is still missing it the release notes. Jira provides a page giving the release notes for each version. I need to regenerate them from the issues on a milestone.

But I'll try to do some actual EasyMock code first... Cheers!

Virtual goods are still mine: Open letter to The Economist

I'm inaugurating a new section that will be more life oriented i.e not talking about software engineering. It might be in English or French depending on my mood and the public expected to read the entry. So this first blog post is more or less an open letter to The Economist.

When I'm not doing software engineering, I'm highly interested in finance, economy and politics. Until recently, I was even a diligent subscriber of The Economist. I've decided two months ago to not renew my subscription. Not because I don't like the magazine anymore but because it was too time consuming to read. I want this year to have the time to read books.

I was receiving the paper version which allows me to have access to the online version. Of course, I'm not keeping the paper versions in my basement since I was having access to them online. I was really surprised to realize that as soon as my subscription expired, I've lost the access to the entire The Economist website content. Event the editions THAT I PAID FOR!

I found that to be quite a cheap decision from a journal as renowned as The Economist.

But I also think it shouldn't be legal. It's been one of my thoughts for a while now.

Virtual properties should have the same "rights" as physical ones. When I buy a physical book, I can sale it, lend it, give it.

If an buy a e-book, I can't do all this. I'm locked in whatever rules the seller wants to apply. And I don't feel good about it. I'm pretty sure I'm not the only one and I'm also pretty sure it creates adoption issues.

Of course you can buy virtual gears. Sellers love to sell you stuff. Virtual or not. But when it's virtual, they currently make sure you will never really be the owner of it.

Yes, I do understand the concept of service. For instance, The Economist offers the website as a service to their customer. They pay the hosting, the development so it makes sense that if I stop paying, the service stops. But it doesn't make sense that I can't download the PDF of all the issues I bought. Of course it means that I would then be allowed to illegally diffuse them. But I can allow photocopy my paper issue. Or scan it.

Meanwhile, I'm left with a bitter taste in my mouth. Until these's a law protecting my virtual properties, I'm not really sure I will trust The Economist anymore.

Finish migrating Objenesis to GitHub

With Codehaus and Google Code closing, I’m happily required to migrate Objenesis and EasyMock.

Both projects have an hybrid hosting using different platforms. I’ve decided to start with Objenesis which is easier to migrate. I’ll describe the process in this post. Hopefully, it will be helpful to someone but, in fact, I’m also looking forward to your feedback to see if I can improve my final setup.

So, Objenesis source code was originally on Google Code. I’ve moved it to GitHub some years ago mainly because pull requests are awesome. The website is also a GitHub page. Part of the documentation is on Google Code wiki, the binaries are on Google Code as well as the issue tracker.

Google being nice folks, they used to provide a nice way to migrate everything to GitHub. But I couldn’t use that because

  • I don’t want the project to end up in henri-tremblay/objenesis. I want it where it is now, in the EasyMock organisation
  • I only want to migrate the wiki and the issues since the sources are already there

So here’s what I did instead.

Issue tracker

The issue tracker was migrated using the IssueExporterTool. It worked perfectly (but is a bit slow as advertised).

Wiki pages

At first, I tried to export the entire Objenesis project to Github to be able to retrieve the wiki pages and then move them to the real source code. The result was quite bad because the tables are not rendered correctly. So I ended up manually migrating each page to markdown. There was only 3 pages so it wasn’t too bad.

Binaries

This was more complicated. Maven binaries are already deployed through the Sonatype Nexus. But I needed to migrate the standalone bundles. I’ve looked into three options:

  • GitHub releases
  • Bintray
  • Both (GitHub releases exported to Bintray)

GitHub releases are created automatically when you tag in git. But you also seem to be able to add some release notes and binaries over that. I didn’t want to dig too much into it. I knew I wanted to be in Bintray in the end. And I wanted easy automation. Bintray was easy to use so I went for Bintray only. Be aware, the way I did the migration is low-tech (but works).

  1. Make a list of all the binaries to migrate
    • Get them with wget https://objenesis.googlecode.com/files/objenesis-xxx.zip
  2. Create an organisation, a distribution repository and an [objenesis package])(https://bintray.com/easymock/distributions/objenesis) in Bintray
  3. Add my GPG key to my Bintray account
  4. Upload everything using REST (curl -T objenesis-xxx.zip -H "X-GPG-PASSPHRASE: ${PASSPHRASE}" -uhenri-tremblay:${API_KEY} https://api.bintray.com/content/easymock/distributions/objenesis/xxx/objenesis-xxx-bin.zip?publish=1)

It works, the only drawback is that no release notes are provided. They’ve always been on the website

Project moved

Finally, I’ve set the “Project Moved” flag to target GitHub. This quite aggressively redirects you to GitHub if you try to access https://code.google.com/p/objenesis.

What I learned today

When creating this blog, the first thing I wanted to talk about are those things you learn everyday when fighting on a technical issue. Because if it happened to me, it will probably happen to you.

What I discover is that a lot of these things are too small to be a blog post or will in fact just repeat a really nice existing blog post where I found my answer in the first place. And a blog post just telling you to read another blog post is not that useful.

Instead, I've decided to tweet these blog posts each time I fell on one. I'll use the #WhatILearnedToday hashtag.  If you're interested, you can then follow me on Twitter.

See ya!