Jupyter notebooks and Python: advantages, limitations, best practices

What are Jupyter notebooks?

The term “Jupyter notebook” can refer to different things: a Jupyter notebook document or the web application that can be used to edit Jupyter notebook documents. This blog post is about the notebook document. This can be used not only in the web application, but also in VS Code, for example.

These are their advantages:

The greatest strength of Jupyter notebooks lies in their structure. They are interactive and consist of a list of input and output blocks. Written code can therefore be broken down into small pieces and executed section by section. The output (e.g., a visualization) can then be seen directly below the code section that generated it. In addition, text blocks can be used to add explanatory content. These functionalities make notebooks particularly interesting for tutorials and initial experiments. Small changes can be easily tested without having to re-execute the entire code.

These are their limitations:

Jupyter notebooks also have two sides. They scale poorly and are difficult to integrate into standard software development processes. But what exactly can go wrong?

• Since the code cells can be executed in any order, they can be used to achieve unwanted behavior. For example, if the visualization is started first and the data is processed only afterwards.
• Notebooks are difficult to break down into individual, reusable modules. This leads to a lot of code duplication and thus to code that is difficult to maintain.
• Since notebooks are technically JSON documents, it is difficult to version the code (e.g., with Git). The large amount of metadata and output blocks makes it difficult to identify the relevant code changes and merge them correctly. In the worst case, collaborative development on a notebook can result in the loss of written code.
• Jupyter notebooks are designed so that all code cells run in one core. This makes it difficult to work asynchronously. If a single line of code runs longer than expected, it blocks all others. This can be a problem, especially with large amounts of data.
• In order to maintain interactive capabilities, the entire notebook must be loaded into memory at all times. This leads to performance losses, especially with large notebooks.