<!-- TOC --><a name="utf8-cpp-utf-8-with-c-in-a-portable-way"></a>

UTF8-CPP: UTF-8 with C++ in a Portable Way

<!-- TOC --><a name="introduction"></a>

Introduction

C++ developers still miss an easy and portable way of handling Unicode encoded strings. The original C++ standard (known as C++98 or C++03) is Unicode agnostic. Some progress has been made in the later editions of the standard, but it is still hard to work with Unicode using only the standard facilities.

I came up with a small, C++98 compatible generic library in order to handle UTF-8 encoded strings. For anybody used to work with STL algorithms and iterators, it should be easy and natural to use. The code is freely available for any purpose - check out the license. The library has been used a lot since the first release in 2006 both in commercial and open-source projects and proved to be stable and useful.

Table of Contents

• UTF8-CPP: UTF-8 with C++ in a Portable Way
  • Introduction
  • Installation
  • Examples of use
    • Introductory Sample
    • Checking if a file contains valid UTF-8 text
    • Ensure that a string contains valid UTF-8 text
  • Points of interest - Design goals and decisions - Alternatives
  • Reference
    • Functions From utf8 Namespace
      • utf8::append
        • octet_iterator append(utfchar32_t cp, octet_iterator result)
        • void append(utfchar32_t cp, std::string& s);
      • utf8::append16
        • word_iterator append16(utfchar32_t cp, word_iterator result)
        • void append(utfchar32_t cp, std::u16string& s)
      • utf8::next
      • utf8::next16
      • utf8::peek_next
      • utf8::prior
      • utf8::advance
      • utf8::distance
      • utf8::utf16to8
        • octet_iterator utf16to8 (u16bit_iterator start, u16bit_iterator end, octet_iterator result)
        • std::string utf16to8(const std::u16string& s)
        • std::string utf16to8(std::u16string_view s)
      • utf8::utf16tou8
        • std::u8string utf16tou8(const std::u16string& s)
        • std::u8string utf16tou8(const std::u16string_view& s)
      • utf8::utf8to16
        • u16bit_iterator utf8to16 (octet_iterator start, octet_iterator end, u16bit_iterator result)
        • std::u16string utf8to16(const std::string& s)
        • std::u16string utf8to16(std::string_view s)
        • std::u16string utf8to16(std::u8string& s)
        • std::u16string utf8to16(std::u8string_view& s)
      • utf8::utf32to8
        • octet_iterator utf32to8 (u32bit_iterator start, u32bit_iterator end, octet_iterator result)
        • std::string utf32to8(const std::u32string& s)
        • std::u8string utf32to8(const std::u32string& s)
        • std::u8string utf32to8(const std::u32string_view& s)
        • std::string utf32to8(const std::u32string& s)
        • std::string utf32to8(std::u32string_view s)
      • utf8::utf8to32
        • u32bit_iterator utf8to32 (octet_iterator start, octet_iterator end, u32bit_iterator result)
        • std::u32string utf8to32(const std::u8string& s)
        • std::u32string utf8to32(const std::u8string_view& s)
        • std::u32string utf8to32(const std::string& s)
        • std::u32string utf8to32(std::string_view s)
      • utf8::find_invalid
        • octet_iterator find_invalid(octet_iterator start, octet_iterator end)
        • const char* find_invalid(const char* str)
        • std::size_t find_invalid(const std::string& s)
        • std::size_t find_invalid(std::string_view s)
      • utf8::is_valid
        • bool is_valid(octet_iterator start, octet_iterator end)
        • bool is_valid(const char* str)
        • bool is_valid(const std::string& s)
        • bool is_valid(std::string_view s)
      • utf8::replace_invalid
        • output_iterator replace_invalid(octet_iterator start, octet_iterator end, output_iterator out, utfchar32_t replacement)
        • std::string replace_invalid(const std::string& s, utfchar32_t replacement)
        • std::string replace_invalid(std::string_view s, char32_t replacement)
      • utf8::starts_with_bom
        • bool starts_with_bom (octet_iterator it, octet_iterator end)
        • bool starts_with_bom(const std::string& s)
        • bool starts_with_bom(std::string_view s)
    • Types From utf8 Namespace
      • utf8::exception
      • utf8::invalid_code_point
      • utf8::invalid_utf8
      • utf8::invalid_utf16
      • utf8::not_enough_room
      • utf8::iterator
        • Member functions
    • Functions From utf8::unchecked Namespace
      • utf8::unchecked::append
      • utf8::unchecked::append16
      • utf8::unchecked::next
      • utf8::next16
      • utf8::unchecked::peek_next
      • utf8::unchecked::prior
      • utf8::unchecked::advance
      • utf8::unchecked::distance
      • utf8::unchecked::utf16to8
      • utf8::unchecked::utf8to16
      • utf8::unchecked::utf32to8
      • utf8::unchecked::utf8to32
      • utf8::unchecked::replace_invalid
    • Types From utf8::unchecked Namespace
      • utf8::iterator
        • Member functions

<!-- TOC end --> <!-- TOC --><a name="installation"></a>

Installation

This is a header-only library and the supported way of deploying it is:

• Download a release from https://github.com/nemtrif/utfcpp/releases into a temporary directory
• Unzip the release
• Copy the content of utfcpp/source file into the directory where you keep include files for your project

The CMakeList.txt file was originally made for testing purposes only, but unfortunately over time I accepted contributions that added install target. This is not a supported way of installing the utfcpp library and I am considering removing the CMakeList.txt in a future release.

<!-- TOC --><a name="examples-of-use"></a>

Examples of use

<!-- TOC --><a name="introductory-sample"></a>

Introductory Sample

To illustrate the use of the library, let's start with a small but complete program that opens a file containing UTF-8 encoded text, reads it line by line, checks each line for invalid UTF-8 byte sequences, and converts it to UTF-16 encoding and back to UTF-8:

#include <fstream>
#include <iostream>
#include <string>
#include <vector>
#include "utf8.h"
using namespace std;
int main(int argc, char** argv)
{
    if (argc != 2) {
        cout << "\nUsage: docsample filename\n";
        return 0;
    }
    const char* test_file_path = argv[1];
    // Open the test file (must be UTF-8 encoded)
    ifstream fs8(test_file_path);
    if (!fs8.is_open()) {
        cout << "Could not open " << test_file_path << endl;
        return 0;
    }

    unsigned line_count = 1;
    string line;
    // Play with all the lines in the file
    while (getline(fs8, line)) {
        // check for invalid utf-8 (for a simple yes/no check, there is also utf8::is_valid function)
#if __cplusplus >= 201103L // C++ 11 or later
        auto end_it = utf8::find_invalid(line.begin(), line.end());
#else
        string::iterator end_it = utf8::find_invalid(line.begin(), line.end());
#endif // C++ 11
        if (end_it != line.end()) {
            cout << "Invalid UTF-8 encoding detected at line " << line_count << "\n";
            cout << "This part is fine: " << string(line.begin(), end_it) << "\n";
        }
        // Get the line length (at least for the valid part)
        int length = utf8::distance(line.begin(), end_it);
        cout << "Length of line " << line_count << " is " << length <<  "\n";

        // Convert it to utf-16
#if __cplusplus >= 201103L // C++ 11 or later
        u16string utf16line = utf8::utf8to16(line);
#else
        vector<unsigned short> utf16line;
        utf8::utf8to16(line.begin(), end_it, back_inserter(utf16line));
#endif // C++ 11
        // And back to utf-8;
#if __cplusplus >= 201103L // C++ 11 or later
        string utf8line = utf8::utf16to8(utf16line);
#else
        string utf8line; 
        utf8::utf16to8(utf16line.begin(), utf16line.end(), back_inserter(utf8line));
#endif // C++ 11
        // Confirm that the conversion went OK:
        if (utf8line != string(line.begin(), end_it))
            cout << "Error in UTF-16 conversion at line: " << line_count << "\n";        

        line_count++;
    } 

    return 0;
}

In the previous code sample, for each line we performed a detection of invalid UTF-8 sequences with find_invalid; the number of characters (more precisely - the number of Unicode code points, including the end of line and even BOM if there is one) in each line was determined with a use of utf8::distance; finally, we have converted each line to UTF-16 encoding with utf8to16 and back to UTF-8 with utf16to8.

Note a different pattern of usage for old compilers. For instance, this is how we convert a UTF-8 encoded string to a UTF-16 encoded one with a pre - C++11 compiler:

    vector<unsigned short> utf16line;
    utf8::utf8to16(line.begin(), end_it, back_inserter(utf16line));

With a more modern compiler, the same operation would look like:

    u16string utf16line = utf8::utf8to16(line);

If __cplusplus macro points to a C++ 11 or later, the library exposes API that takes into account C++ standard Unicode strings and move semantics. With an older compiler, it is still possible to use the same functionality, just in a little less convenient way

In case you do not trust the __cplusplus macro or, for instance, do not want to include the C++ 11 helper functions even with a modern compiler, define UTF_CPP_CPLUSPLUS macro before including utf8.h and assign it a value for the standard you want to use - the values are the same as for the __cplusplus macro. This can be also useful with compilers that are conservative in setting the __cplusplus macro even if they have a good support for a recent standard edition - Microsoft's Visual C++ is one example.

<!-- TOC --><a name="checking-if-a-file-contains-valid-utf-8-text"></a>

Checking if a file contains valid UTF-8 text

Here is a function that checks whether the content of a file is valid UTF-8 encoded text without reading the content into the memory:

bool valid_utf8_file(const char* file_name)
{
    ifstream ifs(file_name);
    if (!ifs)
        return false; // even better, throw here

    istreambuf_iterator<char> it(ifs.rdbuf());
    istreambuf_iterator<char> eos;

    return utf8::is_valid(it, eos);
}

Because the function utf8::is_valid() works with input iterators, we were able to pass an istreambuf_iterator to it and read the content of the file directly without loading it to the memory first.

Note that other functions that take input iterator arguments can be used in a similar way. For instance, to read the content of a UTF-8 encoded text file and convert the text to UTF-16, just do something like:

    utf8::utf8to16(it, eos, back_inserter(u16string));

<!-- TOC --><a name="ensure-that-a-string-contains-valid-utf-8-text"></a>

Ensure that a string contains valid UTF-8