59

i am converting a project from Ant to Maven and i'm having problems with a specific unit test which deals with UTF-8 characters. The problem is about the following String:

String l_string = "ČäÁÓý\n€řЖжЦ\n№ЯФКЛ";

The problem is that the unit test fails, because the String is read as the following:

?äÁÓý
€????
?????

The java class is saved as UTF-8 and i also specify the build encoding to UTF-8 in the pom.xml.

Here is an excerpt of my pom.xml:

...

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

...

<build>
<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.1</version>
        <configuration>
            <source>1.6</source>
            <target>1.6</target>
            <encoding>${project.build.sourceEncoding}</encoding>
        </configuration>
    </plugin>
    <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>2.4</version>
        <configuration>
            <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
        </configuration>
    </plugin>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.15</version>
    </plugin>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-report-plugin</artifactId>
      <version>2.15</version>
    </plugin>
 </plugins>
</build>

Am i missing something here? It would be great, if someone could help me here.

Update

Regarding the test code:

@Test
public void testTransformation()
{

    String l_string = "ČäÁÓý\n€řЖжЦ\n№ЯФКЛ";
    System.out.println( ">>> " + l_string );
     c_log.info( l_string );
    StringBuffer l_stringBuffer = new StringBuffer();
    int l_stringLength = l_string.length();

    String l_fileName = System.getProperty( "user.dir" ) + File.separator + "transformation" + File.separator + "TransformationMap.properties";
    Transformation.init( l_fileName );

    Properties l_props = Transformation.getProps();
    for ( int i = 0; i < l_stringLength; i++ )
    {
        char l_char = l_string.charAt( i );
        int l_intValue = (int) l_char;
        if ( l_intValue <= 255 )
        {
            l_stringBuffer.append( l_char );
        }
        else
        {
            l_stringBuffer.append( l_props.getProperty( String.valueOf( l_char ), "" ) );
        }
    }
    c_log.info( l_stringBuffer.toString() );
    byte[] l_bytes = l_string.getBytes();
    byte[] l_transformedBytes = Transformation.transform( l_bytes );
    assertNotNull( l_transformedBytes );

}

The following logic isn't really relevant(?) because after the first sysout the before mentioned "?" are printed instead of the correct characters (and therefore the following tests fail). There is also no use of a default platform encoding.

The test converts each character according to the TransformationMap.properties file, which is in the following form (just an excerpt):

Ý=Y
ý=y
Ž=Z
ž=z
°=.
€=EUR

It should be noted that the test runs without any problem when i build the project with Ant.

Improve this question
11
  • 1
    What is the test code? Does it use the platform default encoding at any place? Or does the code under test do that somewhere? Commented Jul 15, 2013 at 14:27
  • possible duplicate of Is there a way to make maven build class files with UTF-8 without using the external JAVA_TOOL_OPTIONS? Commented Jul 15, 2013 at 15:04
  • @Joachim Sauer: I updated my posting. Commented Jul 15, 2013 at 15:04
  • @softandsafe: that's not a useful test, because if your output console isn't set to use a unicode encoding, then the output will look wrong, even if l_string contains the correct data (i.e. even if it is compiled correctly). Do you have an actual assert that fails? Or do you just verify visually if it works? Commented Jul 15, 2013 at 15:06
  • @JoachimSauer: I updated my post again. I have an actual assert that fails. Commented Jul 15, 2013 at 15:15

6 Answers 6

Reset to default
159

I have found a "solution" myself:

I had to pass the encoding into the maven-surefire-plugin, but the usual

<encoding>${project.build.sourceEncoding}</encoding>

did not work. I still have no idea why, but when i pass the command line arguments into the plugin, the tests works as they should:

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.15</version>
      <configuration>
        <argLine>${argLine} -Dfile.encoding=${project.build.sourceEncoding}</argLine>
      </configuration>
</plugin>

You need to include ${argLine} - otherwise other plugins that also modify list of arguments, like JaCoCo, will stop working.

Thanks for all your responses and additional comments!

Improve this answer
Sign up to request clarification or add additional context in comments.

15 Comments

windows-1252. It seems to use the OS default encoding, but the encoding is set everywhere in the pom file to UTF-8 even in the surefire-plugin.
Maybe a bit more resilient solution would be <argLine>-Dfile.encoding=${project.build.sourceEncoding}</argLine>
This is still open. Issue moved from codehaus to apache at issues.apache.org/jira/browse/SUREFIRE-951
Saved my day, man! It took me like 2 hours to figure out the root of my problem (tests were running perfectly fine from IntelliJ IDEA, but one test always failed from Maven). The reason was a literal string in Cyrillic was passed properly (as UTF-8) into myBatis mapper when run from IDEA, but wrongly (as Cp-1251) when run with Maven. Adding Surefile plugin configuration into POM file and specifying argline did the trick!
as just commented: This change breaks other functionality. So you must write <argLine>@{argLine} -Dfile.encoding=UTF-8</argLine>
|
10

  1. When debugging Unicode problems, make sure you convert everything to ASCII so you can read and understand what is inside of a String without guesswork. This means you should use, for example, StringEscapeUtils from commons-lang3 to turn ä into \u00e4. That way, you can be sure that you see ? because the console can't print it. And you can distinguish " " (\u0020) from " " (\u00a0)

    In the test case, check the escaped version of the inputs as early as possible to make sure the data is actually what you expect.

    So the code above should be:

    assertEquals("\u010d\u00e4\u....", escape(l_string));

  2. Make sure you use the correct encoding for file I/O. Never use the default encoding of Java, always use InputStreamReader/OutputStreamWriter and specify the encoding to use.

  3. The POM looks correct. Run mvn with -X to make sure it picks up the correct options and runs the Java compiler using the correct options. mvn help:effective-pom might also help.

  4. Disassemble the class file to check the strings. Java will use ? to denote that it couldn't read something.

    If you get the ? from System.out.println( ">>> " + l_string );, this means the code wasn't compiled with UTF-8 or that the source file was maybe saved with another Unicode encoding (UTF-16 or similar).

    Another source of problems could be the properties file. Make sure it was saved with ISO-8859-1 and that it wasn't modified by the compilation process.

  5. Make sure Maven actually compiles your file. Use mvn clean to force a full-recompile.

Improve this answer

Comments

6

I had a really resilient problem of this kind and setting environmental variable

MAVEN_OPTS=-Dfile.encoding=UTF-8

fixed the issue for me.

Improve this answer

Comments

4

Your problem is not the encoding of the source file (and therefore the String inside your class file) but the Problem is the encoding of System.out's implicite PrintStream. It uses file.encoding which represents the System encoding, and this is in Windows the ANSI codepage.

You would have to set up a PrintWriter with the OEM code page (or you use the class which is intended for this: Console).

See also various bugs around this in: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4153167

Improve this answer

Comments

4

this works for me:

...
 <properties>
        **<project.build.sourceEncoding>ISO-8859-1</project.build.sourceEncoding>
        <project.reporting.outputEncoding>ISO-8859-1</project.reporting.outputEncoding>**
    </properties>
...
  <build>
    <finalName>Project</finalName>

    <sourceDirectory>src</sourceDirectory>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>2.3.2</version>
        <configuration>
          <source>1.6</source>
          <target>1.6</target>
          **<encoding>${project.build.sourceEncoding}</encoding>**
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-war-plugin</artifactId>
        <version>2.2</version>
        <configuration>
          <warSourceDirectory>WebContent</warSourceDirectory>
        </configuration>
      </plugin>
    </plugins>
  </build>
Improve this answer

Comments

2

Following works for me without any issues:

       <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>3.1.0</version>
          <configuration>
            <test>com.testsuite.JunitTestSuite</test>
            <argLine>${argLine} -Dfile.encoding=UTF-8</argLine>
          </configuration>
        </plugin>
Improve this answer

1 Comment

This advice (combined with this comment) fixes the bug introduced by this bugfix… sigh

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.