313 lines
13 KiB
ReStructuredText
313 lines
13 KiB
ReStructuredText
Distribution packages, package management, and deploying applications
|
|
=====================================================================
|
|
|
|
Just as the "big" Python, MicroPython supports creation of "third party"
|
|
packages, distributing them, and easily installing them in each user's
|
|
environment. This chapter discusses how these actions are achieved.
|
|
Some familiarity with Python packaging is recommended.
|
|
|
|
Overview
|
|
--------
|
|
|
|
Steps below represent a high-level workflow when creating and consuming
|
|
packages:
|
|
|
|
1. Python modules and packages are turned into distribution package
|
|
archives, and published at the Python Package Index (PyPI).
|
|
2. `upip` package manager can be used to install a distribution package
|
|
on a `MicroPython port` with networking capabilities (for example,
|
|
on the Unix port).
|
|
3. For ports without networking capabilities, an "installation image"
|
|
can be prepared on the Unix port, and transferred to a device by
|
|
suitable means.
|
|
4. For low-memory ports, the installation image can be frozen as the
|
|
bytecode into MicroPython executable, thus minimizing the memory
|
|
storage overheads.
|
|
|
|
The sections below describe this process in details.
|
|
|
|
Distribution packages
|
|
---------------------
|
|
|
|
Python modules and packages can be packaged into archives suitable for
|
|
transfer between systems, storing at the well-known location (PyPI),
|
|
and downloading on demand for deployment. These archives are known as
|
|
*distribution packages* (to differentiate them from Python packages
|
|
(means to organize Python source code)).
|
|
|
|
The MicroPython distribution package format is a well-known tar.gz
|
|
format, with some adaptations however. The Gzip compressor, used as
|
|
an external wrapper for TAR archives, by default uses 32KB dictionary
|
|
size, which means that to uncompress a compressed stream, 32KB of
|
|
contguous memory needs to be allocated. This requirement may be not
|
|
satisfiable on low-memory devices, which may have total memory available
|
|
less than that amount, and even if not, a contiguous block like that
|
|
may be hard to allocate due to memory fragmentation. To accommodate
|
|
these constraints, MicroPython distribution packages use Gzip compression
|
|
with the dictionary size of 4K, which should be a suitable compromise
|
|
with still achieving some compression while being able to uncompressed
|
|
even by the smallest devices.
|
|
|
|
Besides the small compression dictionary size, MicroPython distribution
|
|
packages also have other optimizations, like removing any files from
|
|
the archive which aren't used by the installation process. In particular,
|
|
`upip` package manager doesn't execute ``setup.py`` during installation
|
|
(see below), and thus that file is not included in the archive.
|
|
|
|
At the same time, these optimizations make MicroPython distribution
|
|
packages not compatible with `CPython`'s package manager, ``pip``.
|
|
This isn't considered a big problem, because:
|
|
|
|
1. Packages can be installed with `upip`, and then can be used with
|
|
CPython (if they are compatible with it).
|
|
2. In the other direction, majority of CPython packages would be
|
|
incompatible with MicroPython by various reasons, first of all,
|
|
the reliance on features not implemented by MicroPython.
|
|
|
|
Summing up, the MicroPython distribution package archives are highly
|
|
optimized for MicroPython's target environments, which are highly
|
|
resource constrained devices.
|
|
|
|
|
|
``upip`` package manager
|
|
------------------------
|
|
|
|
MicroPython distribution packages are intended to be installed using
|
|
the `upip` package manager. `upip` is a Python application which is
|
|
usually distributed (as frozen bytecode) with network-enabled
|
|
`MicroPython ports <MicroPython port>`. At the very least,
|
|
`upip` is available in the `MicroPython Unix port`.
|
|
|
|
On any `MicroPython port` providing `upip`, it can be accessed as
|
|
following::
|
|
|
|
import upip
|
|
upip.help()
|
|
upip.install(package_or_package_list, [path])
|
|
|
|
Where *package_or_package_list* is the name of a distribution
|
|
package to install, or a list of such names to install multiple
|
|
packages. Optional *path* parameter specifies filesystem
|
|
location to install under and defaults to the standard library
|
|
location (see below).
|
|
|
|
An example of installing a specific package and then using it::
|
|
|
|
>>> import upip
|
|
>>> upip.install("micropython-pystone_lowmem")
|
|
[...]
|
|
>>> import pystone_lowmem
|
|
>>> pystone_lowmem.main()
|
|
|
|
Note that the name of Python package and the name of distribution
|
|
package for it in general don't have to match, and oftentimes they
|
|
don't. This is because PyPI provides a central package repository
|
|
for all different Python implementations and versions, and thus
|
|
distribution package names may need to be namespaced for a particular
|
|
implementation. For example, all packages from `micropython-lib`
|
|
follow this naming convention: for a Python module or package named
|
|
``foo``, the distribution package name is ``micropython-foo``.
|
|
|
|
For the ports which run MicroPython executable from the OS command
|
|
prompts (like the Unix port), `upip` can be (and indeed, usually is)
|
|
run from the command line instead of MicroPython's own REPL. The
|
|
commands which corresponds to the example above are::
|
|
|
|
micropython -m upip -h
|
|
micropython -m upip install [-p <path>] <packages>...
|
|
micropython -m upip install micropython-pystone_lowmem
|
|
|
|
[TODO: Describe installation path.]
|
|
|
|
|
|
Cross-installing packages
|
|
-------------------------
|
|
|
|
For `MicroPython ports <MicroPython port>` without native networking
|
|
capabilities, the recommend process is "cross-installing" them into a
|
|
"directory image" using the `MicroPython Unix port`, and then
|
|
transferring this image to a device by suitable means.
|
|
|
|
Installing to a directory image involves using ``-p`` switch to `upip`::
|
|
|
|
micropython -m upip install -p install_dir micropython-pystone_lowmem
|
|
|
|
After this command, the package content (and contents of every depenency
|
|
packages) will be available in the ``install_dir/`` subdirectory. You
|
|
would need to transfer contents of this directory (without the
|
|
``install_dir/`` prefix) to the device, at the suitable location, where
|
|
it can be found by the Python ``import`` statement (see discussion of
|
|
the `upip` installation path above).
|
|
|
|
|
|
Cross-installing packages with freezing
|
|
---------------------------------------
|
|
|
|
For the low-memory `MicroPython ports <MicroPython port>`, the process
|
|
described in the previous section does not provide the most efficient
|
|
resource usage,because the packages are installed in the source form,
|
|
so need to be compiled to the bytecome on each import. This compilation
|
|
requires RAM, and the resulting bytecode is also stored in RAM, reducing
|
|
its amount available for storing application data. Moreover, the process
|
|
above requires presence of the filesystem on a device, and the most
|
|
resource-constrained devices may not even have it.
|
|
|
|
The bytecode freezing is a process which resolves all the issues
|
|
mentioned above:
|
|
|
|
* The source code is pre-compiled into bytecode and store as such.
|
|
* The bytecode is stored in ROM, not RAM.
|
|
* Filesystem is not required for frozen packages.
|
|
|
|
Using frozen bytecode requires building the executable (firmware)
|
|
for a given `MicroPython port` from the C source code. Consequently,
|
|
the process is:
|
|
|
|
1. Follow the instructions for a particular port on setting up a
|
|
toolchain and building the port. For example, for ESP8266 port,
|
|
study instructions in ``ports/esp8266/README.md`` and follow them.
|
|
Make sure you can build the port and deploy the resulting
|
|
executable/firmware successfully before proceeding to the next steps.
|
|
2. Build `MicroPython Unix port` and make sure it is in your PATH and
|
|
you can execute ``micropython``.
|
|
3. Change to port's directory (e.g. ``ports/esp8266/`` for ESP8266).
|
|
4. Run ``make clean-frozen``. This step cleans up any previous
|
|
modules which were installed for freezing (consequently, you need
|
|
to skip this step to add additional modules, instead of starting
|
|
from scratch).
|
|
5. Run ``micropython -m upip install -p modules <packages>...`` to
|
|
install packages you want to freeze.
|
|
6. Run ``make clean``.
|
|
7. Run ``make``.
|
|
|
|
After this, you should have the executable/firmware with modules as
|
|
the bytecode inside, which you can deploy the usual way.
|
|
|
|
Few notes:
|
|
|
|
1. Step 5 in the sequence above assumes that the distribution package
|
|
is available from PyPI. If that is not the case, you would need
|
|
to copy Python source files manually to ``modules/`` subdirectory
|
|
of the port port directory. (Note that upip does not support
|
|
installing from e.g. version control repositories).
|
|
2. The firmware for baremetal devices usually has size restrictions,
|
|
so adding too many frozen modules may overflow it. Usually, you
|
|
would get a linking error if this happens. However, in some cases,
|
|
an image may be produced, which is not runnable on a device. Such
|
|
cases are in general bugs, and should be reported and further
|
|
investigated. If you face such a situation, as an initial step,
|
|
you may want to decrease the amount of frozen modules included.
|
|
|
|
|
|
Creating distribution packages
|
|
------------------------------
|
|
|
|
Distribution packages for MicroPython are created in the same manner
|
|
as for CPython or any other Python implementation, see references at
|
|
the end of chapter. Setuptools (instead of distutils) should be used,
|
|
because distutils do not support dependencies and other features. "Source
|
|
distribution" (``sdist``) format is used for packaging. The post-processing
|
|
discussed above, (and pre-processing discussed in the following section)
|
|
is achieved by using custom ``sdist`` command for setuptools. Thus, packaging
|
|
steps remain the same as for the standard setuptools, the user just
|
|
needs to override ``sdist`` command implementation by passing the
|
|
appropriate argument to ``setup()`` call::
|
|
|
|
from setuptools import setup
|
|
import sdist_upip
|
|
|
|
setup(
|
|
...,
|
|
cmdclass={'sdist': sdist_upip.sdist}
|
|
)
|
|
|
|
The sdist_upip.py module as referenced above can be found in
|
|
`micropython-lib`:
|
|
https://github.com/micropython/micropython-lib/blob/master/sdist_upip.py
|
|
|
|
|
|
Application resources
|
|
---------------------
|
|
|
|
A complete application, besides the source code, oftentimes also consists
|
|
of data files, e.g. web page templates, game images, etc. It's clear how
|
|
to deal with those when application is installed manually - you just put
|
|
those data files in the filesystem at some location and use the normal
|
|
file access functions.
|
|
|
|
The situation is different when deploying applications from packages - this
|
|
is more advanced, streamlined and flexible way, but also requires more
|
|
advanced approach to accessing data files. This approach is treating
|
|
the data files as "resources", and abstracting away access to them.
|
|
|
|
Python supports resource access using its "setuptools" library, using
|
|
``pkg_resources`` module. MicroPython, following its usual approach,
|
|
implements subset of the functionality of that module, specifically
|
|
``pkg_resources.resource_stream(package, resource)`` function.
|
|
The idea is that an application calls this function, passing a
|
|
resource identifier, which is a relative path to data file within
|
|
the specified package (usually top-level application package). It
|
|
returns a stream object which can be used to access resource contents.
|
|
Thus, the ``resource_stream()`` emulates interface of the standard
|
|
`open()` function.
|
|
|
|
Implementation-wise, ``resource_stream()`` uses file operations
|
|
underlyingly, if distribution package is install in the filesystem.
|
|
However, it also supports functioning without the underlying filesystem,
|
|
e.g. if the package is frozen as the bytecode. This however requires
|
|
an extra intermediate step when packaging application - creation of
|
|
"Python resource module".
|
|
|
|
The idea of this module is to convert binary data to a Python bytes
|
|
object, and put it into the dictionary, indexed by the resource name.
|
|
This conversion is done automatically using overridden ``sdist`` command
|
|
described in the previous section.
|
|
|
|
Let's trace the complete process using the following example. Suppose
|
|
your application has the following structure::
|
|
|
|
my_app/
|
|
__main__.py
|
|
utils.py
|
|
data/
|
|
page.html
|
|
image.png
|
|
|
|
``__main__.py`` and ``utils.py`` should access resources using the
|
|
following calls::
|
|
|
|
import pkg_resources
|
|
|
|
pkg_resources.resource_stream(__name__, "data/page.html")
|
|
pkg_resources.resource_stream(__name__, "data/image.png")
|
|
|
|
You can develop and debug using the `MicroPython Unix port` as usual.
|
|
When time comes to make a distribution package out of it, just use
|
|
overridden "sdist" command from sdist_upip.py module as described in
|
|
the previous section.
|
|
|
|
This will create a Python resource module named ``R.py``, based on the
|
|
files declared in ``MANIFEST`` or ``MANIFEST.in`` files (any non-``.py``
|
|
file will be considered a resource and added to ``R.py``) - before
|
|
proceeding with the normal packaging steps.
|
|
|
|
Prepared like this, your application will work both when deployed to
|
|
filesystem and as frozen bytecode.
|
|
|
|
If you would like to debug ``R.py`` creation, you can run::
|
|
|
|
python3 setup.py sdist --manifest-only
|
|
|
|
Alternatively, you can use tools/mpy_bin2res.py script from the
|
|
MicroPython distribution, in which can you will need to pass paths
|
|
to all resource files::
|
|
|
|
mpy_bin2res.py data/page.html data/image.png
|
|
|
|
References
|
|
----------
|
|
|
|
* Python Packaging User Guide: https://packaging.python.org/
|
|
* Setuptools documentation: https://setuptools.readthedocs.io/
|
|
* Distutils documentation: https://docs.python.org/3/library/distutils.html
|