split_line is a clean STL string tokenizer written in C++ in less than 100 lines of code. In its simplest form it creates a vector of strings with the tokens from a line of text separated at space, tab, carriage return and newline. In its most complex form it supports user provided delimiters, a user provided quote character, a user provided escape character, a special character for comments and limited abilities to resume tokenization with another part of the string.

• splits a line of text into words delimited by one or more delimiters
• user can provide delimiters (defaults to \t\r\n and space)
• user can provide one special character for quoted text (defaults to „)
• user can provide one special escape character (defaults to \)
• user can provide one special character for comments (disabled by default)
• limited support to resume at another part of the string

• split_line-1.0.zip

int main(int argc, char *argv[]) {
    vector<string> tokens;
    string line = "Writing    programs     \"in C++\"  	is   \
     Fun!!";
 
    split_line(tokens, line);
 
    cout << "Tokens:" << endl;
    for(unsigned int i = 0; i < tokens.size(); i++)
        cout << "'" << tokens[i] << "'" << endl;
 
    return 0;
}

enum {
	SL_NORMAL,
	SL_ESCAPE,
	SL_SAFEMODE,
	SL_SAFEESCAPE,
	SL_COMMENT,
} SPLIT_LINE_STATE;
 
// splits line into tokens and stores them in ret. Supports delimiters, escape characters,
// ignores special characters between safemode_char and between comment_char and line end '\n'.
// returns SPLIT_LINE_STATE the parser was in when returning
int split_line(std::vector<std::string>& ret, std::string& line, const std::string& delimiters = " \t\r\n", char escape_char = '\\', char safemode_char = '"', char comment_char = '\0', int start_state = SL_NORMAL);