Devoxx4kids Material

When living in France, I started to help Devoxx4kids France a little bit. I think it is an amazing idea that will give kids skills that they will definitely needs later.

It also shows them that Facebook on your phone hasn’t appeared by magic. It took a lot of people for hundred of years to discover and master the required technologies.

Now that I live in Montreal, I’ve joined Devoxx4kids Québec. Last time I’ve brought my 3 years old. He played with a LEGO Mindstorm robot he built. I was amazed to see him enjoy it even though Mindstorm are 10+.

Anyway, all this is not the theme of this post :-)

This is a post that will evolve in time where I want to put all the cool learning tools I’ve encountered. And try to comment on them. This is mostly a selfish post that will prevent me from forgetting things. But you might found it useful. So here we go.

First, the official Devoxx4kids material.

Then my personal list (I haven’t tried everything. It’s just a list of cool stuff for me to remember).

Games

  • Robot Turtles: A board game teaching young children how to program without having them notice
  • The Bugs: Another board game teaching young children how to program without having them notice
  • RetroPie: To play old games on a Raspberry Pi

Programming

  • Greenfoot: To program easily little games in Java
  • Scratch: Visual programming games by drag & dropping

Construction

  • LEGO Mindstroms: Lego with captors and engines that you can problem using their interface. Quite complex to create models
  • LEGO WeDo: Like Mindstorms but simpler and for younger kids. The interface is also less buggy
  • LEGO Boost: Available in August 2017. Seems to be an improved version of WeDo.

Electronics

  • littleBits: Really cool bricks to build electronic systems
  • Makey Makey: Fun chipset and electrodes to interact with the computer using everyday objects
  • mBot: Little robot on wheels that you can program using Scratch. Uses a bit too much batteries and in fact can’t do much
  • MakerBloks: Other really cool bricks to build electronic systems
  • Arduino: A full-fledged electronic board to play with

Prevent spontaneous cache creation

UPDATE (2017-02-15): Spring is correctly handling the null from getMissingCache. So you won’t have to overload it. I was misled by a framework that was incorrectly layering the JCacheCacheManager. I stand corrected and the article as well. Thanks to Stéphane Nicoll for his vigilance.

UPDATE (2017-02-16): Spring has a JCacheCacheConfiguration that configure a JCache CacheManager using a list of cache names and a default cache configuration. By default, it uses new MutableConfiguration<>() so the article was updated accordingly.

UPDATE (2017-04-07): My Hibernate PR has been accepted. So you now have an elegant solution with the latest Hibernate versions. The article was updated accordingly.


I played a lot with JCache connectors lately. To plug Ehcache 3 to different things.

I noticed one really dangerous thing. Frameworks tend to spontaneously create caches that were not explicitly defined. I think it is coming from the JCache spirit that there should be a default cache configuration. It is nonetheless a bad idea.

Let’s look at two examples

Spring

Caching in Spring is implemented by spring-cache. To plug JCache to spring-cache you use the JCacheCacheManager. By default when a cache isn’t available in the CacheManager, Spring calls JCacheCacheManager.getMissingCache. So far so good.

The default implementation for this method returns null when a cache doesn’t exist. This null will then be handled at higher levels to throw a nice exception.

If you want to explicitly support spontaneous cache creation, getMissingCache is where you should put your creation code.

However, watch out if you do that. You might lost track of all the existing caches. And please, never do the following.

@Override
protected Cache getMissingCache(String name) {
    Cache cache = super.getMissingCache(name);
    if(cache == null) {
        return new JCacheCache(cacheManager.createCache(name, new MutableConfiguration<>()));
    }
    return cache;
}

It returns a cache configured using default. It is never what you want.

Then, as usual, Spring tries to be nice with us. So if you enable caching (@EnableCaching), that the JSR-107 API is in
the classpath and that you do not expose any CacheManager, Spring will create one for you.

The JCacheCacheConfiguration will get a default JSR-107 CacheManager and add a list of caches taken from the cache property spring.cache.cache-names. These caches are by default created using a new MutableConfiguration<>(). As we said above, this is not a correctly configured cache.

The solution is to expose the wanted cache configuration in a bean.

@Bean
public javax.cache.configuration.Configuration<Object, Object> cacheConfiguration() {
    CacheConfiguration<Object, Object> cacheConfiguration = CacheConfigurationBuilder
        .newCacheConfigurationBuilder(Object.class, Object.class, ResourcePoolsBuilder
            .newResourcePoolsBuilder().heap(100))
        .build();
    javax.cache.configuration.Configuration<Object, Object> configuration =
        Eh107Configuration.fromEhcacheCacheConfiguration(cacheConfiguration);
    return configuration;
}

This bean will be magically used as cache default. You should always do this and never let ``new MutableConfiguration<>()` be used.

Hibernate

To use JCache with Hibernate we need to use the JCacheRegionFactory. The problem with JCacheRegionFactory is that by default if a cache is not found, it will spontaneously create a cache by passing new MutableConfiguration(). This means that instead of using a properly configured cache, you end up with some random default configuration (infinite on heap for Ehcache).

This is really bad because it is pretty hard to detect.

What I recommend in this case is, again, to override the default. In the latest Hibernate versions (5.2.8+) (thanks to https://github.com/hibernate/hibernate-orm/pull/1783[HHH-1783], we can do the following

@Override
protected Cache<Object, Object> createCache(String regionName, Properties properties, CacheDataDescription metadata) {
    throw new IllegalArgumentException("Unknown hibernate cache: " + regionName);
}

In older versions, sadly, there is no straightforward cache creation method to override. The best we have is newDefaultConfig which provides the default configuration. One sad thing is that you don’t have the actual cache name here. You will need to debug to know it.

@Override
protected Configuration<Object, Object> newDefaultConfig(Properties properties, CacheDataDescription metadata) {
    throw new IllegalArgumentException("Unknown hibernate cache: " + metadata);
}

Again, an alternative solution would be to provide a meaningful default cache configuration in this method.

Conclusion

I do understand that frameworks do not like to fail with exceptions. This helps the feeling that they are working out of the box.

But I still think silently not caching or providing random default configuration is dangerous. Using my two workarounds should prevent you a lot of headaches.

Future behaviour through time

Yesterday, I was playing with interruption. At some point I ended up with this code.

public <T> T uninterruptibleGet(Future<T> future) throws ExecutionException {
  while(true) {
    try {
      return future.get();
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
    }
  }
}

This is bad code. Don’t do that. It should be an infinite loop. For those not playing with interruption daily here’s why:

  1. Waiting on get
  2. Be interrupted
  3. Catch the exception and set the interruption state back (Thread.currentThread().interrupt())
  4. Go back to get
  5. Notice we are interrupted (the interruption state is on)
  6. Throw an InterruptedException
  7. Go to 4

The thing is that when running the code, there was no infinite loop.

So I wanted to know why.

The answer is in FutureTask.get().

public V get() throws InterruptedException, ExecutionException {
    int s = state;
    if (s <= COMPLETING) // here
        s = awaitDone(false, 0L);
    return report(s);
}

The get method first look at the state, if done, it just returns. If not, then it interrupts. Here we go, job done.

Then it goes to the Continuous Integration. And goes in an infinite loop. Whaaat???

At first I thought it was some race condition that was happening with other tests (they are run in parallel).

But no. The answer is much simpler than that: JDK 6

The FutureTask code above is from JDK 8. The JDK 6 code is different. It first starts to check the interrupt. So infinite loop it is.

Now, was is the right idiom for an uninterruptibleGet?

To get the long answer, I highly suggest that you go read Java Concurrency in Practice (chapter 7) right now. It is a must read for any Java programmer.

The short answer is this:

public <T> T uninterruptibleGet(Future<T> future) throws ExecutionException {
  boolean interrupted = false;
  try {
    while (true) {
      try {
        return future.get();
      } catch (InterruptedException e) {
        interrupted = true;
      }
    }
  } finally {
    if (interrupted)
      Thread.currentThread().interrupt();
  }
}

EasyMock generic typing

These days, I’m a bit annoyed about Java generics. Because it seems that when you want to be clean, you pretty much always get in troubles.

I’ll give you an example. The method to create a mock with EasyMock is currently typed like this:

public static <T> T mock(Class<T> toMock)

This seems quite straightforward.

But it is also quite annoying to use. For example, let’s try to mock a generic type:

List<String> list = mock(List.class);

It looks like something we would like to easily do but you get this nice warning:

Warning:(51, 33) java: unchecked conversion
  required: java.util.List<java.lang.String>
  found:    java.util.List

And there’s no way to get around it. Here are some attempts:

List<String> list = mock(List<String>.class); // no allowed (won't compile)
List<String> list = EasyMock.<String>mock(List.class); // won't compile either because the parameter type doesn't match)
List<String> list = (List<String>) mock(List.class); // still get a warning

So what’s the way out?

To be less accurate.

Yes. I can’t see any other way out. (Do you?)

So my plan is to change EasyMock typing with this:

public static <T> T mock(Class<?> toMock)

No relation anymore between the parameter and the returned type. WAT!!! Are you crazy?!?

So yes, this will compile without complaints :-( (but I can check the type coherence at runtime)

String list = mock(Integer.class);

But this will now work without any warning :-)

List<String> list = mock(List.class);

You can’t have your cake and eat it it seems. But as usual, I will be happy to be proved wrong.

Gatling Java DSL

My favorite stress tool has been Gatling for many years now.

Scenarios are

  • Easy to construct, It’s just code
  • Easy to version. It’s just code
  • Easy to run. It’s a simple command line that you launch in remote if needed
  • Having great result graphs
  • Easy to datamine. There is a simulation.log file that let’s you do some more digging when needed

My only complaints are

  • You can’t change the running scenario
  • It’s in Scala

The first one might be addressed by Gatling Frontline. I haven’t tried.

The second one is a bit more complicated to solve :-)

In fact, I don’t care that’s in Scala. But I would love to be able to core my scenario in Java.

So just for fun, I did a thought experiment. I’ve coded the Gatling API in Java and then coded the BasicScenario example using this API.

It’s not working for real

The API is just an emply shell. To know how different it would be from the Scala DSL.

The answer is: Not much.

My expectations were that I would need to rely on plenty of Java 8 features to get something close to the Scala DSl. But in fact, not even.

The main differences are:

  • Variables are typed
  • I’ve added a build method. But it’s just for style. I can get rid of it
  • A bunch of semi-colons
  • No syntactic sugar to create a map
  • I’m using Duration is used to time units
  • I need a run method where the setUp method is

That’s quite joyful and probably can be improved.

I’m now wondering if I could have used the Scala classes directly. I’m not good at making Java and Scala interacting.

So if someone is and want to try, please do so.

Manage emails in git

If you are like me, you tend to work on open source using your personal email and at YP using your enterprise email. But git doesn’t handle that well so you need to never forget to do

git config user.email henri.tremblay@somewhere.com

after each git init or git clone.

I finally had the time for work on an almost satisfying solution.

EasyMock 3.4 is out!

This version has finally removed the old (and ugly) partial mocking methods. You are now required to use the partialMockBuilder.

In exchange, you get really nice short methods to create your mocks: mock, niceMock and strictMock.

You also get a better stability since cglib and ASM are now embedded to remove a possible version mismatch with your own dependencies. Note that Objenesis will stay as an explicit dependency.

Change log

Out of SourceForge, all in Bintray

Now that Objenesis is in Bintray. The goal was to also have EasyMock binaries at the same place. They were hosted on SourceForge. And as you might have read, SourceForge is now on the dark side so going out of there was becoming a priority.

This time it was easier. I basically made a list of versions and release dates.

And then I made a loop around the lines to the

  • wget to get the binary and store it keeping the SourceForge structure
  • curl to create the version in Bintray
  • curl to upload the binary
I've manually added the release notes in GitHub but wasn't able to figure out how to push them to Bintray. If someone knows, please tell me.

To be helpful, here is the loop that I used to create the versions.

API_KEY=xxx
PASSPHRASE=xxx

# This is because I want the file upside down to create the older versions first
tail -r $1 > tmp.tsv

while read p; do
version=$(echo "$p" | cut -d' ' -f1)
date=$(echo "$p" | cut -d' ' -f2)

version=${version// /-}

echo $version $date

content="{ \"name\": \"$version\", \"desc\": \"$version\", \"released\": \"${date}T00:00:00.000Z\", \"github_use_tag_release_notes\": false, \"vcs_tag\": \"$version\" }"
echo "$content"

curl -v -XPOST -H "Content-Type: application/json" -H "X-GPG-PASSPHRASE: ${PASSPHRASE}" -uhenri-tremblay:${API_KEY} \
-d "$content" \
https://api.bintray.com/packages/easymock/distributions/easymock/versions

done <tmp.tsv

Migrating EasyMock Jira to GitHub

Following my previous article, I will now describe the Jira migration of EasyMock to GitHub issue tracker.

EasyMock Jira was on Codehaus. So I had to migrate it somewhere else. Instead of migrating to another Jira instance I've decided that EasyMock doesn't need all that complexity anymore and that going to Github would enough.

The migration was made in two steps:

  1. Save everything on my disk
  2. Import everything in GitHub
The first step is important to keep all the original content in case it is needed.

The final result is nice but some things can't be done:
  • Original authors of issues and comments are lost. They are now mentioned in the description but they are not the real authors anymore
  • Creation dates, modification dates, etc are lost
The final code is here. If you need to do something similar it will be useful, however it can't be reused as is since it's not that clean. I also did some parts manually as you will see. 

Export (Export.java)

I used the Spring RestTemplate to retrieve my issues through the Jira API
  • The EASYMOCK project
  • All issues in the project 
  • All fields to keep the meaning of the custom fields
  • All priorities to have the list
  • The project components
  • The project versions
  • The description of all users that have created or fixed an issue
Then I've downloaded all attachments with wget. I've create the list of attachments using a basic shell command (grep attachme issue*.json | grep content | cut -d: -f4). Not pretty but it works.

Everything is on my disk. Now I'm safe.

Import (Import.java)

The import was made in many steps. All could easily automated but it was quicker to just do them manually.

GitHub doesn't have all Jira functionalities. There are no resolution field, affected version,  priority, attachments or custom fields. So things need to be adapted.
  • Affected version is dropped
  • A fixed version becomes a milestone
  • A component is a label
  • A resolution is a label
  • Custom fields are dropped
  • A priority is a label
  • Attachments a pushed in a dedicated repository

Labels

All labels were created manually. However you can look at CreateLabels.java to see how it could be automated.

Milestones

Each Jira version became a milestone. This was done automatically using the Github API. The code is in CreateMilestones.java.

Attachments

GitHub only supports images in attachment. The classical way to do an attachment of something else is to do a gist. I had a look at that and that's why you will see some code to create a gist in GitHubConnector.java.

However, I took another because I had TARs, ZIPs, JARs and other binaries in my attachment. So I've decided to push everything to a dedicated github repository and then reference the files in my issues.

Issues

The final step was to migrate the issues themselves. I had a quick look at the code used to migrate from Google Code to GitHub. There are two importants tricks in there.
  • You can't insert issues too quickly because you can go over the GitHub rate limit
  • You need to put some time between two comments to keep them in order because GitHub sorts the comments according to the timestamps
As soon as you know that, it's basically a mapping from fields to fields. The only final trick is that GitHub issues only have two status: Open and Closed. A newly created issue is always in the Open status. So you then need to modify it to close it if needed.

What is left to do

I'm pretty much done for now. The only thing that is still missing it the release notes. Jira provides a page giving the release notes for each version. I need to regenerate them from the issues on a milestone.

But I'll try to do some actual EasyMock code first... Cheers!

Virtual goods are still mine: Open letter to The Economist

I'm inaugurating a new section that will be more life oriented i.e not talking about software engineering. It might be in English or French depending on my mood and the public expected to read the entry. So this first blog post is more or less an open letter to The Economist.

When I'm not doing software engineering, I'm highly interested in finance, economy and politics. Until recently, I was even a diligent subscriber of The Economist. I've decided two months ago to not renew my subscription. Not because I don't like the magazine anymore but because it was too time consuming to read. I want this year to have the time to read books.

I was receiving the paper version which allows me to have access to the online version. Of course, I'm not keeping the paper versions in my basement since I was having access to them online. I was really surprised to realize that as soon as my subscription expired, I've lost the access to the entire The Economist website content. Event the editions THAT I PAID FOR!

I found that to be quite a cheap decision from a journal as renowned as The Economist.

But I also think it shouldn't be legal. It's been one of my thoughts for a while now.

Virtual properties should have the same "rights" as physical ones. When I buy a physical book, I can sale it, lend it, give it.

If an buy a e-book, I can't do all this. I'm locked in whatever rules the seller wants to apply. And I don't feel good about it. I'm pretty sure I'm not the only one and I'm also pretty sure it creates adoption issues.

Of course you can buy virtual gears. Sellers love to sell you stuff. Virtual or not. But when it's virtual, they currently make sure you will never really be the owner of it.

Yes, I do understand the concept of service. For instance, The Economist offers the website as a service to their customer. They pay the hosting, the development so it makes sense that if I stop paying, the service stops. But it doesn't make sense that I can't download the PDF of all the issues I bought. Of course it means that I would then be allowed to illegally diffuse them. But I can allow photocopy my paper issue. Or scan it.

Meanwhile, I'm left with a bitter taste in my mouth. Until these's a law protecting my virtual properties, I'm not really sure I will trust The Economist anymore.