Java Data Visibility

I was at JCrete two weeks ago. For those who don’t know, it is an awesome Java unconference where everyone with their family to talk about Java. It has been created by Heinz Kabutz a.k.a The Java Specialist. I was really happy to see a bunch of heads I haven’t seen since moving back to Montreal.

One of the sessions I’ve led was about data visibility according to the JMM. I didn’t care about lock and synchronization. Just data visibility.

If I write to this variable, will this other thread see the new value for sure?

I was lucky enough to gather with a handful of subject matter experts.

My goal was to give really simple examples of what works or not. The final code can be found on JCrete’s github.

But I will still describe it here. It’s based on the concept of “Will the thread ever get out of the loop?”. All the examples are almost identical.

@Test
public void test() {
    new Thread(() -> {
        while (!field);
        System.out.println("Done");
    }).start();
    field = true;
}
  1. A thread is started
  2. It loops until a field is flipped to true
  3. The main thread flips the field to true

If the field visibility is correct, when the main thread flips the field, the child thread will see the value and correctly exit the loop. If it’s not, the thread might go in an infinite loop according to the JMM. I’m saying “might” because depending on the CPU architecture, JVM version and the way the wind is blowing, it might see the value correctly on your machine. That doesn’t mean it will work ever after.

I recommend that you first guess the result before looking at the answer.

The first example is a normal field (no final or volatile) without any kind of synchronization.

private boolean stopNormalField;

@Test
public void normalField() {
    new Thread(() -> {
        while (!stopNormalField);
        System.out.println("Done");
    }).start();
    stopNormalField = true;
}

Does it work: No

Data visibility to another thread is not guaranteed by a normal field. The child thread might never see stopNormalField ever value changed.

Now, let’s turn it to a volatile field.

private volatile boolean stopVolatileField;

@Test
public void volatileField() {
    new Thread(() -> {
        while (!stopVolatileField);
        System.out.println("Done");
    }).start();
    stopVolatileField = true;
}

Does it work: Yes

Volatile ensure data visibility. When a volatile field is changed, all other threads are seeing the value right away. Volatile tells the JVM that the value shouldn’t be kept local to the thread.

So far so good.

What about a volatile array?

private volatile boolean[] stopArrayField = new boolean[1];

@Test
public void volatileArrayField() {
    new Thread(() -> {
        while (!stopArrayField[0]);
        System.out.println("Done");
    }).start();
    stopArrayField[0] = true;
}

Does it work: No

Data visibility of array elements is not guaranteed. Elements of a volatile array are not volatile. Yes, it is non-intuitive.

Let’s now start with higher abstraction from java.util.concurrent (JUC). First, an atomic field.

private AtomicBoolean stopAtomicField;

@Test
public void atomicField() {
    stopAtomicField = new AtomicBoolean();
    new Thread(() -> {
        while (!stopAtomicField.get());
        System.out.println("Done");
    }).start();
    stopAtomicField.set(true);
}

Does it work: Yes

Of course it works. Data visibility of the content of an atomic is guaranteed. Note, however, that it doesn’t apply to the field referencing the atomic field (stopAtomicField). It still works on our case because we assign (stopAtomicField = new AtomicBoolean()) before starting the thread. The JVM makes sure a starting thread will see everything that happened-before.

Now. What about synchronization?

@Test
public void synchronizedField() {
    new Thread(() -> {
        while (true) {
            synchronized (this) {
                if(stopNormalField) {
                    break;
                }
            }
        }
        System.out.println("Done");
    }).start();
    synchronized (this) {
        stopNormalField = true;
    }
}

Does it work: Yes

Two threads synchronizing on the same mutex are seeing the same thing inside the synchronized section. All words in bold are really important.

For instance, not synchronizing on the same mutex means all bets are off.

@Test
public void synchronizedOnDifferentMutexField() {
    new Thread(() -> {
        while (true) {
            synchronized (new Object()) { // Wrong mutex
                if(stopNormalField) {
                    break;
                }
            }
        }
        System.out.println("Done");
    }).start();
    synchronized (this) {
        stopNormalField = true;
    }
}

Does it work: No

By the way, watch out for lambda and inner classes. They don’t have the same this. For a lambda, this is the class where the lambda is defined. For an inner class, it is the inner class.

@Test
public void synchronizedInnerClassField() {
    new Thread(new Runnable() {
        @Override
        public void run() {
            while (true) {
                synchronized (DataVisibilityTest.this) { // Specify the this from the outer class
                    if (stopNormalField) {
                        break;
                    }
                }
            }
            System.out.println("Done");
        }
    }).start();
    synchronized (this) {
        stopNormalField = true;
    }
}

Does it work: Yes

Synchronization works. Fair enough. But can I use a lock? I was told locks are better than synchronization.

private Lock lock = new ReentrantLock();

@Test
public void lockField() {
    new Thread(() -> {
        while (true) {
            lock.lock();
            try {
                if(stopNormalField) {
                    break;
                }
            } finally {
                lock.unlock();
            }
        }
        System.out.println("Done");
    }).start();
    lock.lock();
    try {
        stopNormalField = true;
    } finally {
        lock.unlock();
    }
}

Does it work: Yes

Locking provides the same data visibility as synchronizing.

In fact, all JUCs are providing the necessary memory barriers to get the correct visibility. Below, countDown and await will make sure the child thread sees the world correctly.

@Test
public void latchField() {
    CountDownLatch latch = new CountDownLatch(1);
    new Thread(() -> {
        try {
            latch.await();
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        while (!stopNormalField);
        System.out.println("Done");
    }).start();
    stopNormalField = true;
    latch.countDown();
}

Does it work: Yes

As soon as you use JUC abstractions, things tend to magically work in fact. Which is a good thing even though everything magically working is always a bit frightening.

OK. We will finish this post with the Don’t do this at home example. It means it is trickier to get right and you won’t need it for business as usual.

@Test
public void mutexField() {
    new Thread(() -> {
        while (!stopVolatileField);
        while (!stopNormalField);
        System.out.println("Done");
    }).start();
    stopNormalField = true;
    stopVolatileField = true;
}

Does it work: Yes

Here we cause the synchronization of the normal field by using a volatile field. Writing to stopVolatileField causes a happens-before relationship. Happens-before is JMM jargon. It has a script definition. That means what it means in English. That you are sure something happened before. In our case, it means the value written to stopNormalField will be seen by the child thread after it reads stopVolatileField.

Thanks again to all the contributors to this session (you will find some of them on the readme.

Spring GAE Java 8 Objenesis

Objenesis went out last week partly because Google App Engine is now supporting Java 8 in beta. And they are no security manager anymore preventing Objenesis to work.

The problem was that Objenesis was checking if it was running on GAE to pick an instantiating strategy. So it was still running in degraded more on GAE Java 8. It is now fixed.

However, Spring uses an embedded version of Objenesis. So they need to upgrade. Luckily, they are super fast to do so. You can follow the issue but it will be in Spring 4.3.10 and Spring Boot 1.5.5.

Meanwhile, you might be eager to test your shiny new app on this new Google App Engine platform and would be really happy to make it work.

I have a solution for you. It isn’t pretty but it works.

In Java, in case you don’t know, you just need to put your class file in front of another one in the classpath to “shadow” it. It means your class will be used instead of the original implementation. In a war, everything in WEB-INF/classes goes before WEB-INF/lib. So you can easily shadow a class.

That’s what I did. My SpringObjenesis.java version shadows Spring one. It delegates to the real Objenesis implementation instead of the Spring embedded one. You only need to add the latest Objenesis (2.6) as a dependency to your project.

Voilà!

Objenesis 2.6 is out!

An all new shiny Objenesis is out. The framework everybody uses without even knowing it.

It had fun playing with the brand new Google App Engine platform supporting Java 8. They dropped the terrible security manager they used to have to know Objenesis is working perfectly on it. This was brought to my attention by GAE developers. They are now testing mainstream frameworks on their platform to make sure it works. This should make GAE adoption easier.

Java 9 was pretty much already working but I got rid of all the “Illegal accesses”. Not that obvious since in Objenesis, this is how we roll.

Finally, a bit of cleanup in the serialization specification was done. So now a serializing instantiator should instantiate a class by calling the no-arg constructor of the first non-serializable class in the hierarchy. And do nothing else. See the documentation for details.

Change log

Mac UI

I’ve mentioned already why I am now using a mac after years of beloved Windows.

But I’m still complaining about the really bad UI behavior of macOS (I’m on Sierra). And the ca_fr keyboard that should have had the exact same layout that Windows had for years instead of trying to be wiser… But that’s hardware so there’s nothing I can do about it.

During my last round of complains I was ask to write an article about it. So here it is.

First, one of the first thing I’ve fixed is the random behavior of Home and End keys. By default, depending on the app you are in, the behave differently. This is plainly insane. Here is the fix:

cd ~/Library
mkdir KeyBindings
cd KeyBindings
vi DefaultKeyBinding.dict

Put these lines in that file, including the curly braces:

{
/* Remap Home / End keys to be correct */
"\UF729" = "moveToBeginningOfLine:"; /* Home */
"\UF72B" = "moveToEndOfLine:"; /* End */
"$\UF729" = "moveToBeginningOfLineAndModifySelection:"; /* Shift + Home */
"$\UF72B" = "moveToEndOfLineAndModifySelection:"; /* Shift + End */
"^\UF729" = "moveToBeginningOfDocument:"; /* Ctrl + Home */
"^\UF72B" = "moveToEndOfDocument:"; /* Ctrl + End */
"$^\UF729" = "moveToBeginningOfDocumentAndModifySelection:"; /* Shift + Ctrl + Home */
"$^\UF72B" = "moveToEndOfDocumentAndModifySelection:"; /* Shift + Ctrl + End */
}

Then, macOS is lacking Windows docking. So I’ve installed BetterTouchTool for that.

Back to complaining. Here are the things I wasn’t able to fix so far.

  • Cmd+Tab should bring to front the requested window. Always. It doesn’t matter is the window is minimized, in background or whatever. Just bring it in front
  • Cmd+Tab should bring the last used window of an application to the front. Not my 6 Chrome windows. Only the latest one
  • Cmd+ù should behave as Cmd+` when using a ca-fr keyboard. I can probably use the remap trick for that. I tried BTT, it doesn’t work
  • When switching desktop, it shouldn’t matter where my mouse is. Especially to access the dashboard. It should just switch all screens. It there a shortcut to switch screen 1?
  • Cmd+X should cut everywhere. Cmd+C and then Cmd+Alt+V is just not the way my brain works. I think about copy or cut when seeing the file. Not when pasting it. Am I the only one?
  • I use 2 keyboard layouts (US-en and CA-fr). Because of the dumb keyboard. macOS should remember which application is using which keyboard. However, I don’t need it for each Chrome tab

That’s it for the day. Can you help me?

Devoxx4kids Material

When living in France, I started to help Devoxx4kids France a little bit. I think it is an amazing idea that will give kids skills that they will definitely needs later.

It also shows them that Facebook on your phone hasn’t appeared by magic. It took a lot of people for hundred of years to discover and master the required technologies.

Now that I live in Montreal, I’ve joined Devoxx4kids Québec. Last time I’ve brought my 3 years old. He played with a LEGO Mindstorm robot he built. I was amazed to see him enjoy it even though Mindstorm are 10+.

Anyway, all this is not the theme of this post :-)

This is a post that will evolve in time where I want to put all the cool learning tools I’ve encountered. And try to comment on them. This is mostly a selfish post that will prevent me from forgetting things. But you might found it useful. So here we go.

First, the official Devoxx4kids material.

Then my personal list (I haven’t tried everything. It’s just a list of cool stuff for me to remember).

Games

  • Robot Turtles: A board game teaching young children how to program without having them notice
  • The Bugs: Another board game teaching young children how to program without having them notice (note that Magik Square official website seems dead)
  • RetroPie: To play old games on a Raspberry Pi

Programming

  • Greenfoot: To program easily little games in Java
  • Scratch: Visual programming games by drag & dropping

Construction

  • LEGO Mindstroms: Lego with captors and engines that you can problem using their interface. Quite complex to create models
  • LEGO WeDo: Like Mindstorms but simpler and for younger kids. The interface is also less buggy
  • LEGO Boost: Available in August 2017. Seems to be an improved version of WeDo.

Electronics

  • littleBits: Really cool bricks to build electronic systems
  • Makey Makey: Fun chipset and electrodes to interact with the computer using everyday objects
  • mBot: Little robot on wheels that you can program using Scratch. Uses a bit too much batteries and in fact can’t do much
  • MakerBloks: Other really cool bricks to build electronic systems
  • Arduino: A full-fledged electronic board to play with

Prevent spontaneous cache creation

UPDATE (2017-02-15): Spring is correctly handling the null from getMissingCache. So you won’t have to overload it. I was misled by a framework that was incorrectly layering the JCacheCacheManager. I stand corrected and the article as well. Thanks to Stéphane Nicoll for his vigilance.

UPDATE (2017-02-16): Spring has a JCacheCacheConfiguration that configure a JCache CacheManager using a list of cache names and a default cache configuration. By default, it uses new MutableConfiguration<>() so the article was updated accordingly.

UPDATE (2017-04-07): My Hibernate PR has been accepted. So you now have an elegant solution with the latest Hibernate versions. The article was updated accordingly.


I played a lot with JCache connectors lately. To plug Ehcache 3 to different things.

I noticed one really dangerous thing. Frameworks tend to spontaneously create caches that were not explicitly defined. I think it is coming from the JCache spirit that there should be a default cache configuration. It is nonetheless a bad idea.

Let’s look at two examples

Spring

Caching in Spring is implemented by spring-cache. To plug JCache to spring-cache you use the JCacheCacheManager. By default when a cache isn’t available in the CacheManager, Spring calls JCacheCacheManager.getMissingCache. So far so good.

The default implementation for this method returns null when a cache doesn’t exist. This null will then be handled at higher levels to throw a nice exception.

If you want to explicitly support spontaneous cache creation, getMissingCache is where you should put your creation code.

However, watch out if you do that. You might lost track of all the existing caches. And please, never do the following.

@Override
protected Cache getMissingCache(String name) {
    Cache cache = super.getMissingCache(name);
    if(cache == null) {
        return new JCacheCache(cacheManager.createCache(name, new MutableConfiguration<>()));
    }
    return cache;
}

It returns a cache configured using default. It is never what you want.

Then, as usual, Spring tries to be nice with us. So if you enable caching (@EnableCaching), that the JSR-107 API is in
the classpath and that you do not expose any CacheManager, Spring will create one for you.

The JCacheCacheConfiguration will get a default JSR-107 CacheManager and add a list of caches taken from the cache property spring.cache.cache-names. These caches are by default created using a new MutableConfiguration<>(). As we said above, this is not a correctly configured cache.

The solution is to expose the wanted cache configuration in a bean.

@Bean
public javax.cache.configuration.Configuration<Object, Object> cacheConfiguration() {
    CacheConfiguration<Object, Object> cacheConfiguration = CacheConfigurationBuilder
        .newCacheConfigurationBuilder(Object.class, Object.class, ResourcePoolsBuilder
            .newResourcePoolsBuilder().heap(100))
        .build();
    javax.cache.configuration.Configuration<Object, Object> configuration =
        Eh107Configuration.fromEhcacheCacheConfiguration(cacheConfiguration);
    return configuration;
}

This bean will be magically used as cache default. You should always do this and never let ``new MutableConfiguration<>()` be used.

Hibernate

To use JCache with Hibernate we need to use the JCacheRegionFactory. The problem with JCacheRegionFactory is that by default if a cache is not found, it will spontaneously create a cache by passing new MutableConfiguration(). This means that instead of using a properly configured cache, you end up with some random default configuration (infinite on heap for Ehcache).

This is really bad because it is pretty hard to detect.

What I recommend in this case is, again, to override the default. In the latest Hibernate versions (5.2.8+) (thanks to https://github.com/hibernate/hibernate-orm/pull/1783[HHH-1783], we can do the following

@Override
protected Cache<Object, Object> createCache(String regionName, Properties properties, CacheDataDescription metadata) {
    throw new IllegalArgumentException("Unknown hibernate cache: " + regionName);
}

In older versions, sadly, there is no straightforward cache creation method to override. The best we have is newDefaultConfig which provides the default configuration. One sad thing is that you don’t have the actual cache name here. You will need to debug to know it.

@Override
protected Configuration<Object, Object> newDefaultConfig(Properties properties, CacheDataDescription metadata) {
    throw new IllegalArgumentException("Unknown hibernate cache: " + metadata);
}

Again, an alternative solution would be to provide a meaningful default cache configuration in this method.

Conclusion

I do understand that frameworks do not like to fail with exceptions. This helps the feeling that they are working out of the box.

But I still think silently not caching or providing random default configuration is dangerous. Using my two workarounds should prevent you a lot of headaches.

Future behaviour through time

Yesterday, I was playing with interruption. At some point I ended up with this code.

public <T> T uninterruptibleGet(Future<T> future) throws ExecutionException {
  while(true) {
    try {
      return future.get();
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
    }
  }
}

This is bad code. Don’t do that. It should be an infinite loop. For those not playing with interruption daily here’s why:

  1. Waiting on get
  2. Be interrupted
  3. Catch the exception and set the interruption state back (Thread.currentThread().interrupt())
  4. Go back to get
  5. Notice we are interrupted (the interruption state is on)
  6. Throw an InterruptedException
  7. Go to 4

The thing is that when running the code, there was no infinite loop.

So I wanted to know why.

The answer is in FutureTask.get().

public V get() throws InterruptedException, ExecutionException {
    int s = state;
    if (s <= COMPLETING) // here
        s = awaitDone(false, 0L);
    return report(s);
}

The get method first look at the state, if done, it just returns. If not, then it interrupts. Here we go, job done.

Then it goes to the Continuous Integration. And goes in an infinite loop. Whaaat???

At first I thought it was some race condition that was happening with other tests (they are run in parallel).

But no. The answer is much simpler than that: JDK 6

The FutureTask code above is from JDK 8. The JDK 6 code is different. It first starts to check the interrupt. So infinite loop it is.

Now, was is the right idiom for an uninterruptibleGet?

To get the long answer, I highly suggest that you go read Java Concurrency in Practice (chapter 7) right now. It is a must read for any Java programmer.

The short answer is this:

public <T> T uninterruptibleGet(Future<T> future) throws ExecutionException {
  boolean interrupted = false;
  try {
    while (true) {
      try {
        return future.get();
      } catch (InterruptedException e) {
        interrupted = true;
      }
    }
  } finally {
    if (interrupted)
      Thread.currentThread().interrupt();
  }
}

EasyMock generic typing

These days, I’m a bit annoyed about Java generics. Because it seems that when you want to be clean, you pretty much always get in troubles.

I’ll give you an example. The method to create a mock with EasyMock is currently typed like this:

public static <T> T mock(Class<T> toMock)

This seems quite straightforward.

But it is also quite annoying to use. For example, let’s try to mock a generic type:

List<String> list = mock(List.class);

It looks like something we would like to easily do but you get this nice warning:

Warning:(51, 33) java: unchecked conversion
  required: java.util.List<java.lang.String>
  found:    java.util.List

And there’s no way to get around it. Here are some attempts:

List<String> list = mock(List<String>.class); // no allowed (won't compile)
List<String> list = EasyMock.<String>mock(List.class); // won't compile either because the parameter type doesn't match)
List<String> list = (List<String>) mock(List.class); // still get a warning

So what’s the way out?

To be less accurate.

Yes. I can’t see any other way out. (Do you?)

So my plan is to change EasyMock typing with this:

public static <T> T mock(Class<?> toMock)

No relation anymore between the parameter and the returned type. WAT!!! Are you crazy?!?

So yes, this will compile without complaints :-( (but I can check the type coherence at runtime)

String list = mock(Integer.class);

But this will now work without any warning :-)

List<String> list = mock(List.class);

You can’t have your cake and eat it it seems. But as usual, I will be happy to be proved wrong.

Gatling Java DSL

My favorite stress tool has been Gatling for many years now.

Scenarios are

  • Easy to construct, It’s just code
  • Easy to version. It’s just code
  • Easy to run. It’s a simple command line that you launch in remote if needed
  • Having great result graphs
  • Easy to datamine. There is a simulation.log file that let’s you do some more digging when needed

My only complaints are

  • You can’t change the running scenario
  • It’s in Scala

The first one might be addressed by Gatling Frontline. I haven’t tried.

The second one is a bit more complicated to solve :-)

In fact, I don’t care that’s in Scala. But I would love to be able to core my scenario in Java.

So just for fun, I did a thought experiment. I’ve coded the Gatling API in Java and then coded the BasicScenario example using this API.

It’s not working for real

The API is just an emply shell. To know how different it would be from the Scala DSL.

The answer is: Not much.

My expectations were that I would need to rely on plenty of Java 8 features to get something close to the Scala DSl. But in fact, not even.

The main differences are:

  • Variables are typed
  • I’ve added a build method. But it’s just for style. I can get rid of it
  • A bunch of semi-colons
  • No syntactic sugar to create a map
  • I’m using Duration is used to time units
  • I need a run method where the setUp method is

That’s quite joyful and probably can be improved.

I’m now wondering if I could have used the Scala classes directly. I’m not good at making Java and Scala interacting.

So if someone is and want to try, please do so.

Manage emails in git

If you are like me, you tend to work on open source using your personal email and at YP using your enterprise email. But git doesn’t handle that well so you need to never forget to do

git config user.email henri.tremblay@somewhere.com

after each git init or git clone.

I finally had the time for work on an almost satisfying solution.