base-4.20.2.0: Core data structures and operations

Copyright | (c) The University of Glasgow 2008-2009
License | see libraries/base/LICENSE
Maintainer | libraries@haskell.org
Stability | internal
Portability | non-portable
Safe Haskell | Safe
Language | Haskell2010

GHC.IO.Encoding

Description

Text codecs for I/O

The API of this module is unstable and not meant to be consumed by the general public. If you absolutely must depend on it, make sure to use a tight upper bound, e.g., `base < 4.X` rather than `base < 5`, because the interface can change rapidly without much warning.

Synopsis

- data BufferCodec from to state where
  - BufferCodec# {
    - encode# :: CodeBuffer# from to
    - recover# :: Buffer from -> Buffer to -> State# RealWorld -> (# State# RealWorld, Buffer from, Buffer to #)
    - close# :: IO ()
    - getState# :: IO state
    - setState# :: state -> IO ()
    }
  - pattern BufferCodec :: CodeBuffer from to -> (Buffer from -> Buffer to -> IO (Buffer from, Buffer to)) -> IO () -> IO state -> (state -> IO ()) -> BufferCodec from to state
- data TextEncoding = TextEncoding {
  - textEncodingName :: String
  - mkTextDecoder :: IO (TextDecoder dstate)
  - mkTextEncoder :: IO (TextEncoder estate)
  }
- type TextEncoder state = BufferCodec CharBufElem Word8 state
- type TextDecoder state = BufferCodec Word8 CharBufElem state
- data CodingProgress
  - = InputUnderflow
  - | OutputUnderflow
  - | InvalidSequence
- latin1 :: TextEncoding
- latin1_encode :: CharBuffer -> Buffer Word8 -> IO (CharBuffer, Buffer Word8)
- latin1_decode :: Buffer Word8 -> CharBuffer -> IO (Buffer Word8, CharBuffer)
- utf8 :: TextEncoding
- utf8_bom :: TextEncoding
- utf16 :: TextEncoding
- utf16le :: TextEncoding
- utf16be :: TextEncoding
- utf32 :: TextEncoding
- utf32le :: TextEncoding
- utf32be :: TextEncoding
- initLocaleEncoding :: TextEncoding
- getLocaleEncoding :: IO TextEncoding
- getFileSystemEncoding :: IO TextEncoding
- getForeignEncoding :: IO TextEncoding
- setLocaleEncoding :: TextEncoding -> IO ()
- setFileSystemEncoding :: TextEncoding -> IO ()
- setForeignEncoding :: TextEncoding -> IO ()
- char8 :: TextEncoding
- mkTextEncoding :: String -> IO TextEncoding
- argvEncoding :: IO TextEncoding

# Documentation

data BufferCodec from to state Source #

Constructors

BufferCodec# |

Fields

- encode# :: CodeBuffer# from to

  The `encode` function translates elements of the buffer `from` to the buffer `to`. It should translate as many elements as possible given the sizes of the buffers, including translating zero elements if there is either not enough room in `to`, or `from` does not contain a complete multibyte sequence.

  If multiple CodingProgress returns are possible, OutputUnderflow must be preferred to InvalidSequence. This allows GHC's IO library to assume that if we observe InvalidSequence there is at least a single element available in the output buffer.

  The fact that as many elements as possible are translated is used by the IO library in order to report translation errors at the point they actually occur, rather than when the buffer is translated.

- recover# :: Buffer from -> Buffer to -> State# RealWorld -> (# State# RealWorld, Buffer from, Buffer to #)

  The `recover` function is used to continue decoding in the presence of invalid or unrepresentable sequences. This includes both those detected by `encode` returning `InvalidSequence` and those that occur because the input byte sequence appears to be truncated.

  Progress will usually be made by skipping the first element of the `from` buffer. This function should only be called if you are certain that you wish to do this skipping and if the `to` buffer has at least one element of free space. Because this function deals with decoding failure, it assumes that the from buffer has at least one element.

  `recover` may raise an exception rather than skipping anything.

  Currently, some implementations of `recover` may mutate the input buffer. In particular, this feature is used to implement transliteration.

  Since: base-4.4.0.0

- close# :: IO ()

  Resources associated with the encoding may now be released. The `encode` function may not be called again after calling `close`.

- getState# :: IO state

  Return the current state of the codec.

  Many codecs are not stateful, and in these case the state can be represented as `()`. Other codecs maintain a state. For example, UTF-16 recognises a BOM (byte-order-mark) character at the beginning of the input, and remembers thereafter whether to use big-endian or little-endian mode. In this case, the state of the codec would include two pieces of information: whether we are at the beginning of the stream (the BOM only occurs at the beginning), and if not, whether to use the big or little-endian encoding.

- setState# :: state -> IO ()

Bundled Patterns

pattern BufferCodec :: CodeBuffer from to -> (Buffer from -> Buffer to -> IO (Buffer from, Buffer to)) -> IO () -> IO state -> (state -> IO ()) -> BufferCodec from to state |

data TextEncoding Source #

A `TextEncoding` is a specification of a conversion scheme between sequences of bytes and sequences of Unicode characters.

For example, UTF-8 is an encoding of Unicode characters into a sequence of bytes. The `TextEncoding` for UTF-8 is `utf8`.

Constructors

TextEncoding |

Fields

- textEncodingName :: String

  a string that can be passed to `mkTextEncoding` to create an equivalent `TextEncoding`.

- mkTextDecoder :: IO (TextDecoder dstate)

  Creates a means of decoding bytes into characters: the result must not be shared between several byte sequences or simultaneously across threads

- mkTextEncoder :: IO (TextEncoder estate)

  Creates a means of encode characters into bytes: the result must not be shared between several character sequences or simultaneously across threads

#### Instances

Instances details

Show TextEncoding | Since: base-4.3.0.0
Instance details Defined in GHC.Internal.IO.Encoding.Types Methods showsPrec :: Int -> TextEncoding -> ShowS Source # show :: TextEncoding -> String Source # showList :: [TextEncoding] -> ShowS Source #

type TextEncoder state = BufferCodec CharBufElem Word8 state Source #

type TextDecoder state = BufferCodec Word8 CharBufElem state Source #

data CodingProgress Source #

Since: base-4.4.0.0

Constructors

InputUnderflow | Stopped because the input contains insufficient available elements, or all of the input sequence has been successfully translated.
OutputUnderflow | Stopped because the output contains insufficient free elements
InvalidSequence | Stopped because there are sufficient free elements in the output to output at least one encoded ASCII character, but the input contains an invalid or unrepresentable sequence

#### Instances

Instances details

Show CodingProgress | Since: base-4.4.0.0
Instance details Defined in GHC.Internal.IO.Encoding.Types Methods showsPrec :: Int -> CodingProgress -> ShowS Source # show :: CodingProgress -> String Source # showList :: [CodingProgress] -> ShowS Source #
Eq CodingProgress | Since: base-4.4.0.0
Instance details Defined in GHC.Internal.IO.Encoding.Types Methods (==) :: CodingProgress -> CodingProgress -> Bool Source # (/=) :: CodingProgress -> CodingProgress -> Bool Source #

latin1 :: TextEncoding Source #

The Latin1 (ISO8859-1) encoding. This encoding maps bytes directly to the first 256 Unicode code points, and is thus not a complete Unicode encoding. An attempt to write a character greater than `'\255'` to a `Handle` using the `latin1` encoding will result in an error.

latin1_encode :: CharBuffer -> Buffer Word8 -> IO (CharBuffer, Buffer Word8) Source #

latin1_decode :: Buffer Word8 -> CharBuffer -> IO (Buffer Word8, CharBuffer) Source #

utf8 :: TextEncoding Source #

The UTF-8 Unicode encoding

utf8_bom :: TextEncoding Source #

The UTF-8 Unicode encoding, with a byte-order-mark (BOM; the byte sequence 0xEF 0xBB 0xBF). This encoding behaves like `utf8`, except that on input, the BOM sequence is ignored at the beginning of the stream, and on output, the BOM sequence is prepended.

The byte-order-mark is strictly unnecessary in UTF-8, but is sometimes used to identify the encoding of a file.

utf16 :: TextEncoding Source #

The UTF-16 Unicode encoding (a byte-order-mark should be used to indicate endianness).

utf16le :: TextEncoding Source #

The UTF-16 Unicode encoding (little-endian)

utf16be :: TextEncoding Source #

The UTF-16 Unicode encoding (big-endian)

utf32 :: TextEncoding Source #

The UTF-32 Unicode encoding (a byte-order-mark should be used to indicate endianness).

utf32le :: TextEncoding Source #

The UTF-32 Unicode encoding (little-endian)

utf32be :: TextEncoding Source #

The UTF-32 Unicode encoding (big-endian)

initLocaleEncoding :: TextEncoding Source #

Since: base-4.5.0.0

getLocaleEncoding :: IO TextEncoding Source #

The Unicode encoding of the current locale

Since: base-4.5.0.0

getFileSystemEncoding :: IO TextEncoding Source #

The encoding of the current locale, but allowing arbitrary undecodable bytes to be round-tripped through it.

Do not expect the encoding to be Unicode-compatible: it could appear to be ASCII or anything else.

This `TextEncoding` is used to decode and encode command line arguments and environment variables on non-Windows platforms.

On Windows, this encoding *should not* be used if possible because the use of code pages is deprecated: Strings should be retrieved via the "wide" W-family of UTF-16 APIs instead

Since: base-4.5.0.0

getForeignEncoding :: IO TextEncoding Source #

The Unicode encoding of the current locale, but where undecodable bytes are replaced with their closest visual match. Used for the `CString` marshalling functions in Foreign.C.String

Since: base-4.5.0.0

setLocaleEncoding :: TextEncoding -> IO () Source #

Set locale encoding for your program. The locale affects how `Char`s are encoded and decoded when serialized to bytes: e. g., when you read or write files (`readFile'`, `writeFile`) or use standard input/output (`getLine`, `putStrLn`). For instance, if your program prints non-ASCII characters, it is prudent to execute

    setLocaleEncoding utf8

This is necessary, but not enough on Windows, where console is a stateful device, which needs to be configured using `System.Win32.Console.setConsoleOutputCP` and restored back afterwards. These intricacies are covered by code-page package, which offers a crossplatform `System.IO.CodePage.withCodePage` bracket.

Wrong locale encoding typically causes error messages like "invalid argument (cannot decode byte sequence starting from ...)" or "invalid argument (cannot encode character ...)".

Since: base-4.5.0.0

setFileSystemEncoding :: TextEncoding -> IO () Source #

Since: base-4.5.0.0

setForeignEncoding :: TextEncoding -> IO () Source #

Since: base-4.5.0.0

char8 :: TextEncoding Source #

An encoding in which Unicode code points are translated to bytes by taking the code point modulo 256. When decoding, bytes are translated directly into the equivalent code point.

This encoding never fails in either direction. However, encoding discards information, so encode followed by decode is not the identity.

Since: base-4.4.0.0

mkTextEncoding :: String -> IO TextEncoding Source #

Look up the named Unicode encoding. May fail with

- `isDoesNotExistError` if the encoding is unknown

The set of known encodings is system-dependent, but includes at least:

-     UTF-8

- `UTF-16`, `UTF-16BE`, `UTF-16LE`
- `UTF-32`, `UTF-32BE`, `UTF-32LE`

There is additional notation (borrowed from GNU iconv) for specifying how illegal characters are handled:

- a suffix of `//IGNORE`, e.g. `UTF-8//IGNORE`, will cause all illegal sequences on input to be ignored, and on output will drop all code points that have no representation in the target encoding.
- a suffix of `//TRANSLIT` will choose a replacement character for illegal sequences or code points.
- a suffix of `//ROUNDTRIP` will use a PEP383-style escape mechanism to represent any invalid bytes in the input as Unicode codepoints (specifically, as lone surrogates, which are normally invalid in UTF-32). Upon output, these special codepoints are detected and turned back into the corresponding original byte.

In theory, this mechanism allows arbitrary data to be roundtripped via a `String` with no loss of data. In practice, there are two limitations to be aware of:

1. This only stands a chance of working for an encoding which is an ASCII superset, as for security reasons we refuse to escape any bytes smaller than 128. Many encodings of interest are ASCII supersets (in particular, you can assume that the locale encoding is an ASCII superset) but many (such as UTF-16) are not.
2. If the underlying encoding is not itself roundtrippable, this mechanism can fail. Roundtrippable encodings are those which have an injective mapping into Unicode. Almost all encodings meet this criterion, but some do not. Notably, Shift-JIS (CP932) and Big5 contain several different encodings of the same Unicode codepoint.

On Windows, you can access supported code pages with the prefix `CP`; for example, `"CP1250"`.

argvEncoding :: IO TextEncoding Source #

Internal encoding of argv