user3286661 user3286661 - 2 months ago 17
C++ Question

Boost tokenizer re-assignment

I have to read multiple strings from a struct and then tokenize them using boost tokenizer. Basically what I have currently is this:

typedef boost::tokenizer<boost::char_separator<char> > Tokenizer;
boost::char_separator<char> sep(";");

Tokenizer tok1(str1, sep);
...

Tokenizer tok2(str2, sep);
...

Tokenizer tok3(str3, sep);
.....


Note that I am creating a new tokenizer object every time. Is it possible to do this using just one tokenizer object? Something like this:

Tokenizer tok(str1, sep);
...

// tok(str2, sep); or tok = Tokenizer(str2, sep)


PS: I tried both of the above and they failed.

Answer

Use the assign member function to assign a new token source.

#include<iostream>
#include<boost/tokenizer.hpp>
#include<string>

void test(boost::tokenizer<>& tok)
{
    for(boost::tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){
        std::cout << *beg << " : ";
    }
    std::cout << '\n';
}

int main()
{
    std::string s = "This is,  a test";
    boost::tokenizer<> tok(s);

    test(tok);

    tok.assign(s);
    test(tok);

    tok.assign(s);
    test(tok);
}

expected output:

This : is : a : test : 
This : is : a : test : 
This : is : a : test :