Help with Tokenizer Please

Victor Bazarov · *nix forums Guru Joined: 07 Apr 2005 Posts: 3949

electrixnow wrote:

electrixnow · *nix forums beginner Joined: 05 Feb 2006 Posts: 24

I use a code snippet called tokenize that works quite well. But I need
to modify it so it returns an empty string if two commas are found
together: ie. string test="1,2,,4,5"

I would like to have

tokens[0] = "1"
tokens[1] = "2"
tokens[3] = ""
tokens[4] = "4"
tokens[5] = "5"

This would allow me to find missing tokens. The following is the code I
use.
Can anyone show me the modification I would need?
-----------------

void Tokenize(const string& str,
vector<string>& tokens,
const string& delimiters = " ,\t\n")
{
string::size_type lastPos = str.find_first_not_of(delimiters, 0);
// Skip delimiters at beginning.
string::size_type pos = str.find_first_of(delimiters, lastPos);
// Find first "non-delimiter".
while (string::npos != pos || string::npos != lastPos)
{
tokens.push_back(str.substr(lastPos, pos - lastPos));
// Found a token, add it to the vector.
lastPos = str.find_first_not_of(delimiters, pos);
// Skip delimiters. Note the "not_of"
pos = str.find_first_of(delimiters, lastPos);
// Find next "non-delimiter"
}
}

--------------

Thanks in advance,
Grant

Google