|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectgroovy.util.CharsetToolkit
public class CharsetToolkit
Utility class to guess the encoding of a given text file.
Unicode files encoded in UTF-16 (low or big endian) or UTF-8 files with a Byte Order Marker are correctly discovered. For UTF-8 files with no BOM, if the buffer is wide enough, the charset should also be discovered.
A byte buffer of 4KB is usually sufficient to be able to guess the encoding.
Usage:
// guess the encoding Charset guessedCharset = CharsetToolkit.guessEncoding(file, 4096); // create a reader with the correct charset CharsetToolkit toolkit = new CharsetToolkit(file); BufferedReader reader = toolkit.getReader(); // read the file content String line; while ((line = br.readLine())!= null) { System.out.println(line); }
Field Summary | |
---|---|
private byte[] |
buffer
|
private java.nio.charset.Charset |
charset
|
private java.nio.charset.Charset |
defaultCharset
|
private static byte[] |
EMPTY_BYTE_ARRAY
|
private boolean |
enforce8Bit
|
private java.io.File |
file
|
Constructor Summary | |
---|---|
CharsetToolkit(java.io.File file)
Constructor of the CharsetToolkit utility class. |
Method Summary | |
---|---|
static java.nio.charset.Charset[] |
getAvailableCharsets()
Retrieves all the available Charset s on the platform,
among which the default charset . |
java.nio.charset.Charset |
getCharset()
|
java.nio.charset.Charset |
getDefaultCharset()
Retrieves the default Charset |
static java.nio.charset.Charset |
getDefaultSystemCharset()
Retrieve the default charset of the system. |
boolean |
getEnforce8Bit()
Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding. |
java.io.BufferedReader |
getReader()
Gets a BufferedReader (indeed a LineNumberReader ) from the File
specified in the constructor of CharsetToolkit using the charset discovered by the
method guessEncoding() . |
private java.nio.charset.Charset |
guessEncoding()
Guess the encoding of the provided buffer. |
boolean |
hasUTF16BEBom()
Has a Byte Order Marker for UTF-16 Big Endian (utf-16 and ucs-2). |
boolean |
hasUTF16LEBom()
Has a Byte Order Marker for UTF-16 Low Endian (ucs-2le, ucs-4le, and ucs-16le). |
boolean |
hasUTF8Bom()
Has a Byte Order Marker for UTF-8 (Used by Microsoft's Notepad and other editors). |
private static boolean |
isContinuationChar(byte b)
If the byte has the form 10xxxxx, then it's a continuation byte of a multiple byte character; |
private static boolean |
isFiveBytesSequence(byte b)
If the byte has the form 11110xx, then it's the first byte of a five-bytes sequence character. |
private static boolean |
isFourBytesSequence(byte b)
If the byte has the form 11110xx, then it's the first byte of a four-bytes sequence character. |
private static boolean |
isSixBytesSequence(byte b)
If the byte has the form 1110xxx, then it's the first byte of a six-bytes sequence character. |
private static boolean |
isThreeBytesSequence(byte b)
If the byte has the form 1110xxx, then it's the first byte of a three-bytes sequence character. |
private static boolean |
isTwoBytesSequence(byte b)
If the byte has the form 110xxxx, then it's the first byte of a two-bytes sequence character. |
void |
setDefaultCharset(java.nio.charset.Charset defaultCharset)
Defines the default Charset used in case the buffer represents
an 8-bit Charset . |
void |
setEnforce8Bit(boolean enforce)
If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII. |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
private byte[] buffer
private java.nio.charset.Charset defaultCharset
private java.nio.charset.Charset charset
private boolean enforce8Bit
private final java.io.File file
private static final byte[] EMPTY_BYTE_ARRAY
Constructor Detail |
---|
public CharsetToolkit(java.io.File file) throws java.io.IOException
CharsetToolkit
utility class.
file
- of which we want to know the encoding.
java.io.IOException
Method Detail |
---|
public void setDefaultCharset(java.nio.charset.Charset defaultCharset)
Charset
used in case the buffer represents
an 8-bit Charset
.
defaultCharset
- the default Charset
to be returned by guessEncoding()
if an 8-bit Charset
is encountered.public java.nio.charset.Charset getCharset()
public void setEnforce8Bit(boolean enforce)
charset
rather than US-ASCII.
enforce
- a boolean specifying the use or not of US-ASCII.public boolean getEnforce8Bit()
public java.nio.charset.Charset getDefaultCharset()
private java.nio.charset.Charset guessEncoding()
Guess the encoding of the provided buffer.
If Byte Order Markers are encountered at the beginning of the buffer, we immidiately return the charset implied by this BOM. Otherwise, the file would not be a human readable text file.If there is no BOM, this method tries to discern whether the file is UTF-8 or not. If it is not UTF-8, we assume the encoding is the default system encoding (of course, it might be any 8-bit charset, but usually, an 8-bit charset is the default one).
It is possible to discern UTF-8 thanks to the pattern of characters with a multi-byte sequence.
UCS-4 range (hex.) UTF-8 octet sequence (binary) 0000 0000-0000 007F 0xxxxxxx 0000 0080-0000 07FF 110xxxxx 10xxxxxx 0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10xxxxxx 0001 0000-001F FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 0020 0000-03FF FFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 0400 0000-7FFF FFFF 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
With UTF-8, 0xFE and 0xFF never appear.
private static boolean isContinuationChar(byte b)
b
- a byte.
private static boolean isTwoBytesSequence(byte b)
b
- a byte.
private static boolean isThreeBytesSequence(byte b)
b
- a byte.
private static boolean isFourBytesSequence(byte b)
b
- a byte.
private static boolean isFiveBytesSequence(byte b)
b
- a byte.
private static boolean isSixBytesSequence(byte b)
b
- a byte.
public static java.nio.charset.Charset getDefaultSystemCharset()
Charset
.public boolean hasUTF8Bom()
public boolean hasUTF16LEBom()
public boolean hasUTF16BEBom()
public java.io.BufferedReader getReader() throws java.io.FileNotFoundException
BufferedReader
(indeed a LineNumberReader
) from the File
specified in the constructor of CharsetToolkit
using the charset discovered by the
method guessEncoding()
.
BufferedReader
java.io.FileNotFoundException
- if the file is not found.public static java.nio.charset.Charset[] getAvailableCharsets()
Charset
s on the platform,
among which the default charset
.
Charset
s.
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |