Overview

split_line is a clean STL string tokenizer written in C++ in less than 100 lines of code. In its simplest form it creates a vector of strings with the tokens from a line of text separated at space, tab, carriage return and newline. In its most complex form it supports user provided delimiters, a user provided quote character, a user provided escape character, a special character for comments and limited abilities to resume tokenization with another part of the string.

Features

  • splits a line of text into words delimited by one or more delimiters
  • user can provide delimiters (defaults to \t\r\n and space)
  • user can provide one special character for quoted text (defaults to „)
  • user can provide one special escape character (defaults to \)
  • user can provide one special character for comments (disabled by default)
  • limited support to resume at another part of the string

Download

Code Example

int main(int argc, char *argv[]) {
    vector<string> tokens;
    string line = "Writing    programs     \"in C++\"  	is   \
     Fun!!";
 
    split_line(tokens, line);
 
    cout << "Tokens:" << endl;
    for(unsigned int i = 0; i < tokens.size(); i++)
        cout << "'" << tokens[i] << "'" << endl;
 
    return 0;
}

Output:

Tokens:
'Writing'
'programs'
'in C++'
'is'
'Fun!!'

Documentation

A more complex example can be found in cfg_parser in function readFile(). The function resembles a state machine with 5 states (see enum SPLIT_LINE_STATE). It is possible to provide the starting state of the machine which gives you the ability to resume tokenization of a string in some cases. In resuming mode (start_state != SL_NORMAL) the read in characters are appended to the last string in the string vector ret until the state switches back to SL_NORMAL. In cfg_parser this behaviour was used to read in multiline values. However this features does not give you the ability to split a string anywhere yourself and then pass it over to split_line (using the return state as new start_state). The outcome will be different from what you might expect in most cases!

enum {
	SL_NORMAL,
	SL_ESCAPE,
	SL_SAFEMODE,
	SL_SAFEESCAPE,
	SL_COMMENT,
} SPLIT_LINE_STATE;
 
// splits line into tokens and stores them in ret. Supports delimiters, escape characters,
// ignores special characters between safemode_char and between comment_char and line end '\n'.
// returns SPLIT_LINE_STATE the parser was in when returning
int split_line(std::vector<std::string>& ret, std::string& line, const std::string& delimiters = " \t\r\n", char escape_char = '\\', char safemode_char = '"', char comment_char = '\0', int start_state = SL_NORMAL);
State Diagram

Legend

  • character read in / action
  • eat: append the character to the current token
  • finish: append token to token list and start with a new token

License

CC-GNU GPL
This software is licensed under the CC-GNU GPL.