Neomang Neomang - 1 month ago 6x
C Question

strtok() returning NULL when there should be tokens remaining

I'm working on a basic shell and using strtok to break down a line into commands, then commands into arguments. However I'm running into issues with the tokenizer not returning all of the tokens it should.

For example, I feed in the string

ls -l; cat "foo.txt
. The tokenizer should return the commands
ls -l
cat "foo.txt"
. The commands should then be broken into arguments
, and
. However, I get the following as my output.

prompt>ls -l; cat "foo"
Command: ls -l
Number of tokens in command: 2
Token : ls
Token : (null)
Number of tokens in command: 0

My relevant code is below:

char *commands = strtok(line, ";");
int count = 0;

//get # of commands on line
while(commands != NULL){
//printf("Command : %s\n", commands);
commands = strtok(NULL, ";");

commands = strtok(line, ";");
char *command[count];

//build array of commands
for(int i = 0; i < count; i++){
if(commands != NULL){
command[i] = commands;
printf("Command: %s\n", command[i]);
commands = strtok(NULL, ";");

//Fork Loop
for(int i = 0; i < count; i++){

//printf("Command: %s\n", command[i]);
char *arglist = strtok(command[i], " ");
int arglistc = 0;

//Count number of args in command
while(arglist != NULL){
arglist = strtok(NULL, " ");

printf("Number of tokens in command: %d\n", arglistc);

char *args[arglistc];
arglist = strtok(command[i], " ");

//Build array of args
for(int j = 0; j < arglistc; j++){
args[i] = arglist;
printf("Arglist value : %s\n", arglist);
printf("Token : %s\n", args[i]);
arglist = strtok(NULL, " ");

I'm not sure what I'm doing wrong as I looked up how to use strtok to populate an array and I'm doing just as the solution instructed.


The problem

strtok modifies the string you tokenize, replacing delimiters it finds with 0. The result is a number of strings stored embedded in the original array for your string.

Solution 1: Don't modify the array

strchr will find the first occurrence of a character, we can use it to count the number of tokens. Just don't increment the count on loops where the following character is a delimiter. Then you may use it again (or strtok) when you want to iterate over actual tokens.

You may also use strpbrk if you want to allow multiple delimiter options.

Solution 2: Traverse the strings embedded in the array

Start the "Token:" loop at command[i] and move to strtok(arglist + strlen(arglist) + 1, " ") each step.


This is C, there are of course other solutions.