Accessing Nested Files in our Python App

The Problem:

Python by default cannot find a nested file.

When our Python project starts to get more complex we will inevetably write multiple .py files and then we will start to organize our .py files into directories and sub-directories.

Now Python has an import statement which allows us to reference a module (a .py file) from another module. The import statement will allow us to bring a variable, a function, a class or the total contents of a file into another file. Below is a table of contents for this tutorial. So if you are just skimming for an answer to 'nested files' please just jump down to the __init__.py section.

Table of Contents:

1. Basic import
2. A look at our file structure
3. Introducing __init__.py
4. Empty __init__.py files
5. Code in one __init__.py file
6. Code in both __init__.py files

And as a side note I really like the syntax and readability of Pythons implementation of requiring files/modules. I believe it is one of the better implementations in the industry.

1. Basic import

  # bring the whole object
  import os             

  # only bring a specific item
  from time import timezone  

  # we can rename or 'alias' our import
  from myFile import function as fn  

    

So I mentioned above in the sub-title that by default Pythons import statement cannot find a nested file. But before we get to how to make Python find our files lets look at what Python can recognize as a default.

1. Standard Library Modules
2. Any file that is on the same "level" as it is.

The standard library is all the prewritten code that comes with the stock Python language. These are items like math functionality, operating system functions, time and date methods, etc. Most all the common operations that you would expect a programming language to have. The second item is just saying that whatever is on the same 'level' (read here not nested) as the file being called.

2. A look at our file structure

    /myApp
        |- main.py
        |
        |- myFile.py
        |
        |- /Dir1
              |- file1.py
              |
              |- /Dir2
                    |- file2.py

      * NOTE: a forward slash '/' indicates a directory
  

Inside our main.py file.

  # main.py

  import sys           # ok -> this is part of standard library
  from time import timezone # ok -> this is part of standard library
  import myFile        # ok -> this is on the same 'level'
  import file1         # fail -> ModuleNotFoundError: No module named 'file1'

  

The first three lines in main.py will execute fine but the fourth line will throw an error because Python only knows to look in the two places we spoke about above.

So before we go into the solution lets define some terms:

Blocks: a piece of Python code that will be executed as a unit. An Example is: a module, a function body and class definitions. Please note here that a module can be a .py file that we define.

Module: An organizational unit of Python code. Modules have a namespace and are loaded by Python via importing.

Package: A module with a __path__ attribute and can contain submodules or subpackages.

3. Introducing __init__.py

Python has two types of packages, regular packages and namespace packages. So a regular package is created by putting a __init__.py file in a directory. And when the regular package is imported into a file the __init__.py is executed. But even though this __init__.py file is executed one thing that is sort of strange is that this file can be left empty of contents (an empty file).

Also __init__.py file will require that the files listed in it are loaded in the order that they are listed, which is great if you would like more fine grained control. This can be very helpful if one file is retrieving data or providing setup that is used by the second file.

4. Empty __init__.py files

      /myApp
          |- main.py
          |
          |- myFile.py
          |
          |- /Dir1
                |- __init__.py
                |
                |- file1.py
                |
                |- /Dir2
                      |- __init__.py
                      |
                      |- file2.py

    

So what we have done here is add a file named __init__.py into each directory that has no contents, it is an empty file. Although we can (and will in a little bit) add some contents to our __init__.py, this empty file is perfectly valid to Python.

So with an empty __init__.py file here is how we can use it.

  # file1.py

  def firstFn():
    print('Hello from file1.py')

  def secondFn():
    print('Hello again from file1.py')

  

  # file2.py

  def fnFile2():
    print('Hello from file2.py')

  def secondFnFile2():
    print('Hello again from file2.py')

  

  # main.py

  import Dir1.file1

  Dir1.file1.firstFn()    # prints 'Hello from file1.py'
  Dir1.file1.secondFn()   # prints 'Hello again from file1.py'

  

Or we can do this ...

  # main.py

  from Dir1 import file1
  from Dir1.Dir2 import file2

  file1.firstFn()         # prints 'Hello from file1.py'
  file1.secondFn()        # prints 'Hello again from file1.py'

  file2.fnFile2()         # prints 'Hello from file2.py'
  file2.secondFnFile2()   # prints 'Hello again from file2.py'

  

The first thing to point out is we are using a dot notation . to seperate our elements. Which allows us to 'chain' the directory to file to function. Although this is verbose and some people may not like the length of having to write everything out, it does provide clarity to the statement and does not try to hide anything.

5. Code in one __init__.py file

  # __init__.py

  from .file1 import firstFn, secondFn
  from .Dir2 import file2

  

Note that we have the above code only in our Dir1/__init__.py file and our Dir2/__init__.py file is still empty.

  # main.py

  from Dir1 import firstFn, secondFn, file2

  firstFn()         # prints 'Hello from file1.py'
  secondFn()        # prints 'Hello again from file1.py'

  file2.fnFile2()         # prints 'Hello from file2.py'
  file2.secondFnFile2()   # prints 'Hello again from file2.py'

  

6. Code in both __init__.py files

For our last example we alter the second line in the Dir1/__init__.py file and add some code to Dir2/__init__.py file.

  # __init__.py

  from .file1 import firstFn, secondFn
  from .Dir2 import fnFile2, secondFnFile2

  

  # __init__.py

  from .file2 import fnFile2, secondFnFile2

  

  # main.py

  from Dir1 import firstFn, secondFn, fnFile2, secondFnFile2

  firstFn()   # prints 'Hello from file1.py'
  secondFn()  # prints 'Hello again from file1.py'
  fnFile2()   # prints 'Hello from file2.py'
  secondFnFile2()  # prints 'Hello again from file2.py'

    

So with this last example you can see that our result is being able to call each function from its nested file without having to use 'method chaining' and it gives us a much cleaner look. Although I would argue that it introduces some magic in the sense that it hides some of the complexity. But this is the balance that you need to decide on being a software engineer.

Thank you for reading this far and HAPPY CODING!!!