Isabelle Tan Isabelle Tan - 10 days ago 7
Python Question

Pybrain reinforcement learning; dimension of state

I am working on a project to combine reinforcement learning with traffic light simulations using the package Pybrain. I have read the tutorial and implemented my own subclasses of

Environment
and
Task
. I am using an
ActionValueNetwork
as controller because I want my state to be a vector with continuous values such that it can contain information about for example the number of cars waiting on each lane, the total waiting times per lane and more.

I set the input dimensions of the
ActionValueNetwork
to the dimensions of my state vector, which would suggest that it's possible to use vectors as state variables. When I use the Q-learner or the SARSA learner the code runs fine at first but I obtain an error message as soon as the method
learn()
is called. This function contains the line

state = int(state)


and the error message is

TypeError: only length-1 arrays can be converted to Python scalars


which would suggest that only scalar shaped states can be used.

Does the pybrain reinforcement learning environment support vector shaped states? If so, how can I modify my code such that it will work with their implementations of Q-learning or other methods?

Answer

I have found my problem in the meantime. I was using the Q() learner but should have used the NFQ() learner which implements the neural-fitted Q-learning algorithm. Now it works.