veich veich - 1 month ago 11
Python Question

Python imports for machine learning

I'm switching from R to Python for new machine learning project I'm doing.

I found one code architecture which seems ok (image below) but I have problems with with implementation if I want to keep my code DRY.

enter image description here

So basically I have a piece of code (functions) that I want to share across different modules (namely

data
,
features
and
model
s located under
src
module).
That code in this case contains functions for loading and saving the data.

The problem is that I cannot import functions from upper module (for example if I had
functions.py
directly under
src
folder)

I could add separate
functions.py
inside every module (
data
,
features
,
models
) but that would defeat the purpose of me trying to create well structured project.

I assume I'm not the only one with this issue.

How is this usually handled?

How do you create shared module that can be used by all most deeply nested modules? (I'd like to import the same module into
make_dataset.py
,
build_features.py
,
predict_model.py
,
train_model.py
)

EDIT:

I also forgot to mention I use virtualenv and global space should not be contaminated.

Answer

You need to add your project to your system path so that python can find the module.

This can be done within python:

import sys
sys.path.insert(0, fullPathToProjectFolder)

Then you will be able to call your modules with

import general_functions