mirrors
/
micropython
mirror of https://github.com/micropython/micropython.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

top: Replace upip with mip everywhere. 

Updates all README.md and docs, and manifests to `require("mip")`.

Also extend and improve the documentation on freezing and packaging.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit is contained in:
Jim Mussared 2022-09-29 17:49:58 +10:00
parent ba3652f15d
commit 924a3e03ec
16 changed files with 404 additions and 878 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docs/develop/gettingstarted.rst
View File

@ -322,7 +322,8 @@ tests
tools tools
Contains helper tools including the ``upip`` and the ``pyboard.py`` module. Contains scripts used by the build and CI process, as well as user tools such
as ``pyboard.py`` and ``mpremote``.
examples examples

2
docs/develop/optimizations.rst
View File

@ -25,6 +25,8 @@ into the firmware image as part of the main firmware compilation process, which
the bytecode will be executed from ROM. This can lead to a significant memory saving, and the bytecode will be executed from ROM. This can lead to a significant memory saving, and
reduce heap fragmentation. reduce heap fragmentation.
See :ref:`manifest` for more information.
Variables Variables
--------- ---------

2
docs/esp8266/tutorial/intro.rst
View File

@ -23,7 +23,7 @@ convertor to make the UART available to your PC.
The minimum requirement for flash size is 1Mbyte. There is also a special The minimum requirement for flash size is 1Mbyte. There is also a special
build for boards with 512KB, but it is highly limited comparing to the build for boards with 512KB, but it is highly limited comparing to the
normal build: there is no support for filesystem, and thus features which normal build: there is no support for filesystem, and thus features which
depend on it won't work (WebREPL, upip, etc.). As such, 512KB build will depend on it won't work (WebREPL, mip, etc.). As such, 512KB build will
be more interesting for users who build from source and fine-tune parameters be more interesting for users who build from source and fine-tune parameters
for their particular application. for their particular application.

2
docs/pyboard/tutorial/lcd160cr_skin.rst
View File

@ -42,7 +42,7 @@ There is a test program which you can use to test the features of the display,
and which also serves as a basis to start creating your own code that uses the and which also serves as a basis to start creating your own code that uses the
LCD. This test program is available on GitHub LCD. This test program is available on GitHub
`here <https://github.com/micropython/micropython/blob/master/drivers/display/lcd160cr_test.py>`__. `here <https://github.com/micropython/micropython/blob/master/drivers/display/lcd160cr_test.py>`__.
Copy it to the board over USB mass storage, or by using `mpremote`. Copy it to the board over USB mass storage, or by using :ref:`mpremote`.
To run the test from the MicroPython prompt do:: To run the test from the MicroPython prompt do::

30
docs/reference/glossary.rst
View File

@ -52,7 +52,7 @@ Glossary
cross-compiler cross-compiler
Also known as ``mpy-cross``. This tool runs on your PC and converts a Also known as ``mpy-cross``. This tool runs on your PC and converts a
:term:`.py file` containing MicroPython code into a :term:`.mpy file` :term:`.py file` containing MicroPython code into a :term:`.mpy file`
containing MicroPython bytecode. This means it loads faster (the board containing MicroPython :term:`bytecode`. This means it loads faster (the board
doesn't have to compile the code), and uses less space on flash (the doesn't have to compile the code), and uses less space on flash (the
bytecode is more space efficient). bytecode is more space efficient).
@ -128,7 +128,7 @@ Glossary
Unlike the :term:`CPython` stdlib, micropython-lib modules are Unlike the :term:`CPython` stdlib, micropython-lib modules are
intended to be installed individually - either using manual copying or intended to be installed individually - either using manual copying or
using :term:`upip`. using :term:`mip`.
MicroPython port MicroPython port
MicroPython supports different :term:`boards <board>`, RTOSes, and MicroPython supports different :term:`boards <board>`, RTOSes, and
@ -151,16 +151,26 @@ Glossary
machine-independent features. It can also function in a similar way to machine-independent features. It can also function in a similar way to
:term:`CPython`'s ``python`` executable. :term:`CPython`'s ``python`` executable.
mip
A package installer for MicroPython (mip - "mip installs packages"). It
installs MicroPython packages either from :term:`micropython-lib`,
GitHub, or arbitrary URLs. mip can be used on-device on
network-capable boards, and internally by tools such
as :term:`mpremote`.
mpremote
A tool for interacting with a MicroPython device. See :ref:`mpremote`.
.mpy file .mpy file
The output of the :term:`cross-compiler`. A compiled form of a The output of the :term:`cross-compiler`. A compiled form of a
:term:`.py file` that contains MicroPython bytecode instead of Python :term:`.py file` that contains MicroPython :term:`bytecode` instead of
source code. Python source code.
native native
Usually refers to "native code", i.e. machine code for the target Usually refers to "native code", i.e. machine code for the target
microcontroller (such as ARM Thumb, Xtensa, x86/x64). The ``@native`` microcontroller (such as ARM Thumb, Xtensa, x86/x64). The ``@native``
decorator can be applied to a MicroPython function to generate native decorator can be applied to a MicroPython function to generate native
code instead of bytecode for that function, which will likely be code instead of :term:`bytecode` for that function, which will likely be
faster but use more RAM. faster but use more RAM.
port port
@ -193,8 +203,10 @@ Glossary
as a serial port over USB. as a serial port over USB.
upip upip
(Literally, "micro pip"). A package manager for MicroPython, inspired A now-obsolete package manager for MicroPython, inspired
by :term:`CPython`'s pip, but much smaller and with reduced by :term:`CPython`'s pip, but much smaller and with reduced
functionality. functionality. See its replacement, :term:`mip`.
upip runs both on the :term:`Unix port <MicroPython Unix port>` and on
:term:`baremetal` ports which offer filesystem and networking support. webrepl
A way of connecting to the REPL (and transferring files) on a device
over the internet from a browser. See https://micropython.org/webrepl

271
docs/reference/manifest.rst
View File

@ -1,35 +1,177 @@
.. _manifest:
MicroPython manifest files MicroPython manifest files
========================== ==========================
When building firmware for a device the following components are included in Summary
the compilation process: -------
- the core MicroPython virtual machine and runtime MicroPython has a feature that allows Python code to be "frozen" into the
- port-specific system code and drivers to interface with the firmware, as an alternative to loading code from the filesystem.
microcontroller/device that the firmware is targeting
- standard built-in modules, like ``sys``
- extended built-in modules, like ``json`` and ``machine``
- extra modules written in C/C++
- extra modules written in Python
All the modules included in the firmware are available via ``import`` from This has the following benefits:
Python code. The extra modules written in Python that are included in a build
(the last point above) are called *frozen modules*, and are specified by a
``manifest.py`` file. Changing this manifest requires rebuilding the firmware.
It's also possible to add additional modules to the filesystem of the device - the code is pre-compiled to bytecode, avoiding the need for the Python
once it is up and running. Adding and removing modules to/from the filesystem source to be compiled at load-time.
does not require rebuilding the firmware so is a simpler process than rebuilding - the bytecode can be executed directly from ROM (i.e. flash memory) rather than
firmware. The benefit of using a manifest is that frozen modules are more being copied into RAM. Similarly any constant objects (strings, tuples, etc)
efficient: they are faster to import and take up less RAM once imported. are loaded from ROM also. This can lead to significantly more memory being
available for your application.
- on devices that do not have a filesystem, this is the only way to
load Python code.
MicroPython manifest files are Python files and can contain arbitrary Python During development, freezing is generally not recommended as it will
code. There are also a set of commands (predefined functions) which are used significantly slow down your development cycle, as each update will require
to specify the Python source files to include. These commands are described re-flashing the entire firmware. However, it can still be useful to
below. selectively freeze some rarely-changing dependencies (such as third-party
libraries).
Freezing source code The way to list the Python files to be be frozen into the firmware is via
-------------------- a "manifest", which is a Python file that will be interpreted by the build
process. Typically you would write a manifest file as part of a board
definition, but you can also write a stand-alone manifest file and use it with
an existing board definition.
Manifest files can define dependencies on libraries from :term:`micropython-lib`
as well as Python files on the filesystem, and also on other manifest files.
Writing manifest files
----------------------
A manifest file is a Python file containing a series of function calls. See the
available functions defined below.
Any paths used in manifest files can include the following variables. These all
resolve to absolute paths.
- ``$(MPY_DIR)`` -- path to the micropython repo.
- ``$(MPY_LIB_DIR)`` -- path to the micropython-lib submodule. Prefer to use
``require()``.
- ``$(PORT_DIR)`` -- path to the current port (e.g. ``ports/stm32``)
- ``$(BOARD_DIR)`` -- path to the current board
(e.g. ``ports/stm32/boards/PYBV11``)
Custom manifest files should not live in the main MicroPython repository. You
should keep them in version control with the rest of your project.
Typically a manifest used for compiling firmware will need to include the port
manifest, which might include frozen modules that are required for the board to
function. If you just want to add additional modules to an existing board, then
include the board manifest (which will in turn include the port manifest).
Building with a custom manifest
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Your manifest can be specified on the ``make`` command line with:
.. code-block:: bash
$ make BOARD=MYBOARD FROZEN_MANIFEST=/path/to/my/project/manifest.py
This applies to all ports, including CMake-based ones (e.g. esp32, rp2), as the
Makefile wrapper that will pass this into the CMake build.
Adding a manifest to a board definition
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you have a custom board definition, you can make it include your custom
manifest automatically. On make-based ports (most ports), in your
``mpconfigboard.mk`` set the ``FROZEN_MANIFEST`` variable.
.. code-block:: makefile
FROZEN_MANIFEST ?= $(BOARD_DIR)/manifest.py
On CMake-based ports (e.g. esp32, rp2), instead use ``mpconfigboard.cmake``
.. code-block:: cmake
set(MICROPY_FROZEN_MANIFEST ${MICROPY_BOARD_DIR}/manifest.py)
High-level functions
~~~~~~~~~~~~~~~~~~~~
Note: The ``opt`` keyword argument can be set on the various functions, this controls
the optimisation level used by the cross-compiler.
See :func:`micropython.opt_level`.
.. function:: package(package_path, files=None, base_path=".", opt=None)
This is equivalent to copying the "package_path" directory to the device
(except as frozen code).
In the simplest case, to freeze a package "foo" in the current directory:
.. code-block:: python3
package("foo")
will recursively include all .py files in foo, and will be frozen as
``foo/**/*.py``.
If the package isn't in the same directory as the manifest file, use ``base_path``:
.. code-block:: python3
package("foo", base_path="path/to/libraries")
You can use the variables above, such as ``$(PORT_DIR)`` in ``base_path``.
To restrict to certain files in the package use ``files`` (note: paths
should be relative to the package): ``package("foo", files=["bar/baz.py"])``.
.. function:: module(module_path, base_path=".", opt=None)
Include a single Python file as a module.
If the file is in the current directory:
.. code-block:: python3
module("foo.py")
Otherwise use base_path to locate the file:
.. code-block:: python3
module("foo.py", base_path="src/drivers")
You can use the variables above, such as ``$(PORT_DIR)`` in ``base_path``.
.. function:: require(name, unix_ffi=False)
Require a package by name (and its dependencies) from :term:`micropython-lib`.
Optionally specify unix_ffi=True to use a module from the unix-ffi directory.
.. function:: include(manifest_path)
Include another manifest.
Typically a manifest used for compiling firmware will need to include the
port manifest, which might include frozen modules that are required for
the board to function.
The *manifest* argument can be a string (filename) or an iterable of
strings.
Relative paths are resolved with respect to the current manifest file.
If the path is to a directory, then it implicitly includes the
manifest.py file inside that directory.
You can use the variables above, such as ``$(PORT_DIR)`` in ``manifest_path``.
.. function:: metadata(description=None, version=None, license=None, author=None)
Define metadata for this manifest file. This is useful for manifests for
micropython-lib packages.
Low-level functions
~~~~~~~~~~~~~~~~~~~
These functions are documented for completeness, but with the exception of
``freeze_as_str`` all functionality can be accessed via the high-level functions.
.. function:: freeze(path, script=None, opt=0) .. function:: freeze(path, script=None, opt=0)
@ -42,9 +184,7 @@ Freezing source code
module will start after *path*, i.e. *path* is excluded from the module module will start after *path*, i.e. *path* is excluded from the module
name. name.
If *path* is relative, it is resolved to the current ``manifest.py``. Use If *path* is relative, it is resolved to the current ``manifest.py``.
``$(MPY_DIR)``, ``$(MPY_LIB_DIR)``, ``$(PORT_DIR)``, ``$(BOARD_DIR)`` if you
need to access specific paths.
If *script* is None, all files in *path* will be frozen. If *script* is None, all files in *path* will be frozen.
@ -75,71 +215,48 @@ Freezing source code
Freeze the input, which must be ``.mpy`` files that are frozen directly. Freeze the input, which must be ``.mpy`` files that are frozen directly.
See ``freeze()`` for further details on the arguments. See ``freeze()`` for further details on the arguments.
Including other manifest files
------------------------------
.. function:: include(manifest, **kwargs)
Include another manifest.
The *manifest* argument can be a string (filename) or an iterable of
strings.
Relative paths are resolved with respect to the current manifest file.
Optional *kwargs* can be provided which will be available to the included
script via the *options* variable.
For example:
.. code-block:: python3
include("path.py", extra_features=True)
then in path.py:
.. code-block:: python3
options.defaults(standard_features=True)
# freeze minimal modules.
if options.standard_features:
# freeze standard modules.
if options.extra_features:
# freeze extra modules.
Examples Examples
-------- --------
To freeze a single file which is available as ``import mydriver``, use: To freeze a single file from the current directory which will be available as
``import mydriver``, use:
.. code-block:: python3 .. code-block:: python3
freeze(".", "mydriver.py") module("mydriver.py")
To freeze a set of files which are available as ``import test1`` and To freeze a directory of files in a subdirectory "mydriver" of the current
``import test2``, and which are compiled with optimisation level 3, use: directory which will be available as ``import mydriver``, use:
.. code-block:: python3 .. code-block:: python3
freeze("/path/to/tests", ("test1.py", "test2.py"), opt=3) package("mydriver")
To freeze a module which can be imported as ``import mymodule``, use: To freeze the "hmac" library from :term:`micropython-lib`, use:
.. code-block:: python3 .. code-block:: python3
freeze( require("hmac")
"../relative/path",
(
"mymodule/__init__.py",
"mymodule/core.py",
"mymodule/extra.py",
),
)
To include a manifest from the MicroPython repository, use: A more complete example of a custom ``manifest.py`` file for the ``PYBD_SF2``
board is:
.. code-block:: python3 .. code-block:: python3
include("$(MPY_DIR)/extmod/uasyncio/manifest.py") # Include the board's default manifest.
include("$(BOARD_DIR)/manifest.py")
# Add a custom driver
module("mydriver.py")
# Add aiorepl from micropython-lib
require("aiorepl")
Then the board can be compiled with
.. code-block:: bash
$ cd ports/stm32
$ make BOARD=PYBD_SF2 FROZEN_MANIFEST=~/src/myproject/manifest.py
Note that most boards do not have their own ``manifest.py``, rather they use the
port one directly, in which case your manifest should just
``include("$(PORT_DIR)/boards/manifest.py")`` instead.

463
docs/reference/packages.rst
View File

@ -1,314 +1,153 @@
.. _packages: .. _packages:
Distribution packages, package management, and deploying applications Package management
===================================================================== ==================
Just as the "big" Python, MicroPython supports creation of "third party" Installing packages with ``mip``
packages, distributing them, and easily installing them in each user's --------------------------------
environment. This chapter discusses how these actions are achieved.
Some familiarity with Python packaging is recommended. Network-capable boards include the ``mip`` module, which can install packages
from :term:`micropython-lib` and from third-party sites (including GitHub).
Overview
-------- ``mip`` ("mip installs packages") is similar in concept to Python's ``pip`` tool,
however it does not use the PyPI index, rather it uses :term:`micropython-lib`
Steps below represent a high-level workflow when creating and consuming as its index by default. ``mip`` will automatically fetch compiled
packages: :term:`.mpy file` when downloading from micropython-lib.
1. Python modules and packages are turned into distribution package The most common way to use ``mip`` is from the REPL::
archives, and published at the Python Package Index (PyPI).
2. :term:`upip` package manager can be used to install a distribution package >>> import mip
on a :term:`MicroPython port` with networking capabilities (for example, >>> mip.install("pkgname") # Installs the latest version of "pkgname" (and dependencies)
on the Unix port). >>> mip.install("pkgname", version="x.y") # Installs version x.y of "pkgname"
3. For ports without networking capabilities, an "installation image" >>> mip.install("pkgname", mpy=False) # Installs the source version (i.e. .py rather than .mpy files)
can be prepared on the Unix port, and transferred to a device by
suitable means. ``mip`` will detect an appropriate location on the filesystem by searching
4. For low-memory ports, the installation image can be frozen as the ``sys.path`` for the first entry ending in ``/lib``. You can override the
bytecode into MicroPython executable, thus minimizing the memory destination using ``target``, but note that this path must be in ``sys.path`` to be
storage overheads. able to subsequently import it.::
The sections below describe this process in details. >>> mip.install("pkgname", target="third-party")
>>> sys.path.append("third-party")
Distribution packages
--------------------- As well as downloading packages from the micropython-lib index, ``mip`` can also
install third-party libraries. The simplest way is to download a file directly::
Python modules and packages can be packaged into archives suitable for
transfer between systems, storing at the well-known location (PyPI), >>> mip.install("http://example.com/x/y/foo.py")
and downloading on demand for deployment. These archives are known as >>> mip.install("http://example.com/x/y/foo.mpy")
*distribution packages* (to differentiate them from Python packages
(means to organize Python source code)). When installing a file directly, the ``target`` argument is still supported to set
the destination path, but ``mpy`` and ``version`` are ignored.
The MicroPython distribution package format is a well-known tar.gz
format, with some adaptations however. The Gzip compressor, used as The URL can also start with ``github:`` as a simple way of pointing to content
an external wrapper for TAR archives, by default uses 32KB dictionary hosted on GitHub::
size, which means that to uncompress a compressed stream, 32KB of
contiguous memory needs to be allocated. This requirement may be not >>> mip.install("github:org/repo/path/foo.py") # Uses default branch
satisfiable on low-memory devices, which may have total memory available >>> mip.install("github:org/repo/path/foo.py", version="branch-or-tag") # Optionally specify the branch or tag
less than that amount, and even if not, a contiguous block like that
may be hard to allocate due to memory fragmentation. To accommodate More sophisticated packages (i.e. with more than one file, or with dependencies)
these constraints, MicroPython distribution packages use Gzip compression can be downloaded by specifying the path to their ``package.json``.
with the dictionary size of 4K, which should be a suitable compromise
with still achieving some compression while being able to uncompressed >>> mip.install("http://example.com/x/package.json")
even by the smallest devices. >>> mip.install("github:org/user/path/package.json")
Besides the small compression dictionary size, MicroPython distribution If no json file is specified, then "package.json" is implicitly added::
packages also have other optimizations, like removing any files from
the archive which aren't used by the installation process. In particular, >>> mip.install("http://example.com/x/")
:term:`upip` package manager doesn't execute ``setup.py`` during installation >>> mip.install("github:org/repo")
(see below), and thus that file is not included in the archive. >>> mip.install("github:org/repo", version="branch-or-tag")
At the same time, these optimizations make MicroPython distribution
packages not compatible with :term:`CPython`'s package manager, ``pip``. Using ``mip`` on the Unix port
This isn't considered a big problem, because: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Packages can be installed with :term:`upip`, and then can be used with On the Unix port, ``mip`` can be used at the REPL as above, and also by using ``-m``::
CPython (if they are compatible with it).
2. In the other direction, majority of CPython packages would be $ ./micropython -m mip install pkgname-or-url
incompatible with MicroPython by various reasons, first of all, $ ./micropython -m mip install pkgname-or-url@version
the reliance on features not implemented by MicroPython.
The ``--target=path``, ``--no-mpy``, and ``--index`` arguments can be set::
Summing up, the MicroPython distribution package archives are highly
optimized for MicroPython's target environments, which are highly $ ./micropython -m mip install --target=third-party pkgname
resource constrained devices. $ ./micropython -m mip install --no-mpy pkgname
$ ./micropython -m mip install --index https://host/pi pkgname
``upip`` package manager Installing packages with ``mpremote``
------------------------ -------------------------------------
MicroPython distribution packages are intended to be installed using The :term:`mpremote` tool also includes the same functionality as ``mip`` and
the :term:`upip` package manager. :term:`upip` is a Python application which is can be used from a host PC to install packages to a locally connected device
usually distributed (as frozen bytecode) with network-enabled (e.g. via USB or UART)::
:term:`MicroPython ports <MicroPython port>`. At the very least,
:term:`upip` is available in the :term:`MicroPython Unix port`. $ mpremote install pkgname
$ mpremote install pkgname@x.y
On any :term:`MicroPython port` providing :term:`upip`, it can be accessed as $ mpremote install http://example.com/x/y/foo.py
following:: $ mpremote install github:org/repo
$ mpremote install github:org/repo@branch-or-tag
import upip
upip.help() The ``--target=path``, ``--no-mpy``, and ``--index`` arguments can be set::
upip.install(package_or_package_list, [path])
$ mpremote install --target=/flash/third-party pkgname
Where *package_or_package_list* is the name of a distribution $ mpremote install --no-mpy pkgname
package to install, or a list of such names to install multiple $ mpremote install --index https://host/pi pkgname
packages. Optional *path* parameter specifies filesystem
location to install under and defaults to the standard library Installing packages manually
location (see below). ----------------------------
An example of installing a specific package and then using it:: Packages can also be installed (in either .py or .mpy form) by manually copying
the files to the device. Depending on the board this might be via USB Mass Storage,
>>> import upip the :term:`mpremote` tool (e.g. ``mpremote fs cp path/to/package.py :package.py``),
>>> upip.install("micropython-pystone_lowmem") :term:`webrepl`, etc.
[...]
>>> import pystone_lowmem Writing & publishing packages
>>> pystone_lowmem.main() -----------------------------
Note that the name of Python package and the name of distribution Publishing to :term:`micropython-lib` is the easiest way to make your package
package for it in general don't have to match, and oftentimes they broadly accessible to MicroPython users, and automatically available via
don't. This is because PyPI provides a central package repository ``mip`` and ``mpremote`` and compiled to bytecode. See
for all different Python implementations and versions, and thus https://github.com/micropython/micropython-lib for more information.
distribution package names may need to be namespaced for a particular
implementation. For example, all packages from `micropython-lib` To write a "self-hosted" package that can be downloaded by ``mip`` or
follow this naming convention: for a Python module or package named ``mpremote``, you need a static webserver (or GitHub) to host either a
``foo``, the distribution package name is ``micropython-foo``. single .py file, or a package.json file alongside your .py files.
For the ports which run MicroPython executable from the OS command A typical package.json for an example ``mlx90640`` library looks like::
prompts (like the Unix port), `upip` can be (and indeed, usually is)
run from the command line instead of MicroPython's own REPL. The {
commands which corresponds to the example above are:: "urls": [
["mlx90640/__init__.py", "github:org/micropython-mlx90640/mlx90640/__init__.py"],
micropython -m upip -h ["mlx90640/utils.py", "github:org/micropython-mlx90640/mlx90640/utils.py"]
micropython -m upip install [-p <path>] <packages>... ],
micropython -m upip install micropython-pystone_lowmem "deps": [
["collections-defaultdict", "latest"],
[TODO: Describe installation path.] ["os-path", "latest"]
],
"version": "0.2"
Cross-installing packages }
-------------------------
This includes two files, hosted at a GitHub repo named
For :term:`MicroPython ports <MicroPython port>` without native networking ``org/micropython-mlx90640``, which install into the ``mlx90640`` directory on
capabilities, the recommend process is "cross-installing" them into a the device. It depends on ``collections-defaultdict`` and ``os-path`` which will
"directory image" using the :term:`MicroPython Unix port`, and then be installed automatically.
transferring this image to a device by suitable means.
Freezing packages
Installing to a directory image involves using ``-p`` switch to :term:`upip`:: -----------------
micropython -m upip install -p install_dir micropython-pystone_lowmem When a Python module or package is imported from the device filesystem, it is
compiled into :term:`bytecode` in RAM, ready to be executed by the VM. For
After this command, the package content (and contents of every dependency a :term:`.mpy file`, this conversion has been done already, but the bytecode
packages) will be available in the ``install_dir/`` subdirectory. You still ends up in RAM.
would need to transfer contents of this directory (without the
``install_dir/`` prefix) to the device, at the suitable location, where For low-memory devices, or for large applications, it can be advantageous to
it can be found by the Python ``import`` statement (see discussion of instead run the bytecode from ROM (i.e. flash memory). This can be done
the :term:`upip` installation path above). by "freezing" the bytecode into the MicroPython firmware, which is then flashed
to the device. The runtime performance is the same (although importing is
faster), but it can free up significant amounts of RAM for your program to
Cross-installing packages with freezing use.
---------------------------------------
The downside of this approach is that it's much slower to develop, because you
For the low-memory :term:`MicroPython ports <MicroPython port>`, the process have to flash the firmware each time, but it can be still useful to freeze
described in the previous section does not provide the most efficient dependencies that don't change often.
resource usage,because the packages are installed in the source form,
so need to be compiled to the bytecome on each import. This compilation Freezing is done by writing a manifest file and using it in the build, often as
requires RAM, and the resulting bytecode is also stored in RAM, reducing part of a custom board definition. See the :ref:`manifest` guide for more
its amount available for storing application data. Moreover, the process information.
above requires presence of the filesystem on a device, and the most
resource-constrained devices may not even have it.
The bytecode freezing is a process which resolves all the issues
mentioned above:
* The source code is pre-compiled into bytecode and store as such.
* The bytecode is stored in ROM, not RAM.
* Filesystem is not required for frozen packages.
Using frozen bytecode requires building the executable (firmware)
for a given :term:`MicroPython port` from the C source code. Consequently,
the process is:
1. Follow the instructions for a particular port on setting up a
toolchain and building the port. For example, for ESP8266 port,
study instructions in ``ports/esp8266/README.md`` and follow them.
Make sure you can build the port and deploy the resulting
executable/firmware successfully before proceeding to the next steps.
2. Build :term:`MicroPython Unix port` and make sure it is in your PATH and
you can execute ``micropython``.
3. Change to port's directory (e.g. ``ports/esp8266/`` for ESP8266).
4. Run ``make clean-frozen``. This step cleans up any previous
modules which were installed for freezing (consequently, you need
to skip this step to add additional modules, instead of starting
from scratch).
5. Run ``micropython -m upip install -p modules <packages>...`` to
install packages you want to freeze.
6. Run ``make clean``.
7. Run ``make``.
After this, you should have the executable/firmware with modules as
the bytecode inside, which you can deploy the usual way.
Few notes:
1. Step 5 in the sequence above assumes that the distribution package
is available from PyPI. If that is not the case, you would need
to copy Python source files manually to ``modules/`` subdirectory
of the port directory. (Note that upip does not support
installing from e.g. version control repositories).
2. The firmware for baremetal devices usually has size restrictions,
so adding too many frozen modules may overflow it. Usually, you
would get a linking error if this happens. However, in some cases,
an image may be produced, which is not runnable on a device. Such
cases are in general bugs, and should be reported and further
investigated. If you face such a situation, as an initial step,
you may want to decrease the amount of frozen modules included.
Creating distribution packages
------------------------------
Distribution packages for MicroPython are created in the same manner
as for CPython or any other Python implementation, see references at
the end of chapter. Setuptools (instead of distutils) should be used,
because distutils do not support dependencies and other features. "Source
distribution" (``sdist``) format is used for packaging. The post-processing
discussed above, (and pre-processing discussed in the following section)
is achieved by using custom ``sdist`` command for setuptools. Thus, packaging
steps remain the same as for the standard setuptools, the user just
needs to override ``sdist`` command implementation by passing the
appropriate argument to ``setup()`` call::
from setuptools import setup
import sdist_upip
setup(
...,
cmdclass={'sdist': sdist_upip.sdist}
)
The sdist_upip.py module as referenced above can be found in
`micropython-lib`:
https://github.com/micropython/micropython-lib/blob/master/sdist_upip.py
Application resources
---------------------
A complete application, besides the source code, oftentimes also consists
of data files, e.g. web page templates, game images, etc. It's clear how
to deal with those when application is installed manually - you just put
those data files in the filesystem at some location and use the normal
file access functions.
The situation is different when deploying applications from packages - this
is more advanced, streamlined and flexible way, but also requires more
advanced approach to accessing data files. This approach is treating
the data files as "resources", and abstracting away access to them.
Python supports resource access using its "setuptools" library, using
``pkg_resources`` module. MicroPython, following its usual approach,
implements subset of the functionality of that module, specifically
``pkg_resources.resource_stream(package, resource)`` function.
The idea is that an application calls this function, passing a
resource identifier, which is a relative path to data file within
the specified package (usually top-level application package). It
returns a stream object which can be used to access resource contents.
Thus, the ``resource_stream()`` emulates interface of the standard
`open()` function.
Implementation-wise, ``resource_stream()`` uses file operations
underlyingly, if distribution package is install in the filesystem.
However, it also supports functioning without the underlying filesystem,
e.g. if the package is frozen as the bytecode. This however requires
an extra intermediate step when packaging application - creation of
"Python resource module".
The idea of this module is to convert binary data to a Python bytes
object, and put it into the dictionary, indexed by the resource name.
This conversion is done automatically using overridden ``sdist`` command
described in the previous section.
Let's trace the complete process using the following example. Suppose
your application has the following structure::
my_app/
__main__.py
utils.py
data/
page.html
image.png
``__main__.py`` and ``utils.py`` should access resources using the
following calls::
import pkg_resources
pkg_resources.resource_stream(__name__, "data/page.html")
pkg_resources.resource_stream(__name__, "data/image.png")
You can develop and debug using the :term:`MicroPython Unix port` as usual.
When time comes to make a distribution package out of it, just use
overridden "sdist" command from sdist_upip.py module as described in
the previous section.
This will create a Python resource module named ``R.py``, based on the
files declared in ``MANIFEST`` or ``MANIFEST.in`` files (any non-``.py``
file will be considered a resource and added to ``R.py``) - before
proceeding with the normal packaging steps.
Prepared like this, your application will work both when deployed to
filesystem and as frozen bytecode.
If you would like to debug ``R.py`` creation, you can run::
python3 setup.py sdist --manifest-only
Alternatively, you can use tools/mpy_bin2res.py script from the
MicroPython distribution, in which can you will need to pass paths
to all resource files::
mpy_bin2res.py data/page.html data/image.png
References
----------
* Python Packaging User Guide: https://packaging.python.org/
* Setuptools documentation: https://setuptools.readthedocs.io/
* Distutils documentation: https://docs.python.org/3/library/distutils.html

2
examples/hwapi/README.md
View File

@ -116,7 +116,7 @@ For example, one may invent a "configuration manager" helper module which will
try to detect current board (among well-known ones), and load appropriate try to detect current board (among well-known ones), and load appropriate
`hwconfig_*.py` - this assumes that a user would lazily deploy them all `hwconfig_*.py` - this assumes that a user would lazily deploy them all
(or that application will be automatically installed, e.g. using MicroPython's (or that application will be automatically installed, e.g. using MicroPython's
`upip` package manager). The key point in this case remains the same as `mip` package manager). The key point in this case remains the same as
elaborated above - always assume there can, and will be a custom configuration, elaborated above - always assume there can, and will be a custom configuration,
and it should be well supported. So, any automatic detection should be and it should be well supported. So, any automatic detection should be
overridable by a user, and instructions how to do so are among the most overridable by a user, and instructions how to do so are among the most

3
ports/esp32/boards/manifest.py
View File

@ -1,11 +1,10 @@
freeze("$(PORT_DIR)/modules") freeze("$(PORT_DIR)/modules")
module("upip.py", base_path="$(MPY_DIR)/tools", opt=3)
module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)
include("$(MPY_DIR)/extmod/uasyncio") include("$(MPY_DIR)/extmod/uasyncio")
# Require some micropython-lib modules. # Require some micropython-lib modules.
require("dht") require("dht")
require("ds18x20") require("ds18x20")
require("mip")
require("neopixel") require("neopixel")
require("ntptime") require("ntptime")
require("onewire") require("onewire")

18
ports/esp8266/README.md
View File

@ -200,20 +200,22 @@ Python prompt over WiFi, connecting through a browser.
- GitHub repository https://github.com/micropython/webrepl. - GitHub repository https://github.com/micropython/webrepl.
Please follow the instructions there. Please follow the instructions there.
__upip__ __mip__
The ESP8266 port comes with builtin `upip` package manager, which can The ESP8266 port comes with the built-in `mip` package manager, which can
be used to install additional modules (see the main README for more be used to install additional modules:
information):
``` ```
>>> import upip >>> import mip
>>> upip.install("micropython-pystone_lowmem") >>> mip.install("hmac")
[...] [...]
>>> import pystone_lowmem >>> import hmac
>>> pystone_lowmem.main() >>> hmac.new(b"1234567890", msg="hello world").hexdigest()
``` ```
See [Package management](https://docs.micropython.org/en/latest/reference/packages.html) for more
information about `mip`.
Downloading and installing packages may requite a lot of free memory, Downloading and installing packages may requite a lot of free memory,
if you get an error, retry immediately after the hard reset. if you get an error, retry immediately after the hard reset.

7
ports/esp8266/boards/manifest.py
View File

@ -1,9 +1,8 @@
freeze("$(PORT_DIR)/modules") freeze("$(PORT_DIR)/modules")
module("upip.py", base_path="$(MPY_DIR)/tools", opt=3)
module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)
require("ntptime")
require("dht") require("dht")
require("onewire")
require("ds18x20") require("ds18x20")
require("mip")
require("neopixel") require("neopixel")
require("ntptime")
require("onewire")
require("webrepl") require("webrepl")

4
ports/rp2/boards/PICO_W/manifest.py
View File

@ -1,7 +1,5 @@
include("../manifest.py") include("../manifest.py")
module("upip.py", base_path="$(MPY_DIR)/tools", opt=3) require("mip")
module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)
require("ntptime") require("ntptime")
require("urequests") require("urequests")

26
ports/unix/README.md
View File

@ -24,21 +24,26 @@ Use `CTRL-D` (i.e. EOF) to exit the shell.
Learn about command-line options (in particular, how to increase heap size Learn about command-line options (in particular, how to increase heap size
which may be needed for larger applications): which may be needed for larger applications):
$ ./micropython -h $ ./build-standard/micropython -h
To run the complete testsuite, use: To run the complete testsuite, use:
$ make test $ make test
The Unix port comes with a builtin package manager called upip, e.g.: The Unix port comes with a built-in package manager called `mip`, e.g.:
$ ./micropython -m upip install micropython-pystone $ ./build-standard/micropython -m mip install hmac
$ ./micropython -m pystone
Browse available modules on or
[PyPI](https://pypi.python.org/pypi?%3Aaction=search&term=micropython).
Standard library modules come from the $ ./build-standard/micropython
[micropython-lib](https://github.com/micropython/micropython-lib) project. >>> import mip
>>> mip.install("hmac")
Browse available modules at [micropython-lib]
(https://github.com/micropython/micropython-lib). See
[Package management](https://docs.micropython.org/en/latest/reference/packages.html)
for more information about `mip`.
External dependencies External dependencies
--------------------- ---------------------
@ -65,6 +70,5 @@ or not). If you intend to build MicroPython with additional options
(like cross-compiling), the same set of options should be passed to `make (like cross-compiling), the same set of options should be passed to `make
deplibs`. To actually enable/disable use of dependencies, edit the deplibs`. To actually enable/disable use of dependencies, edit the
`ports/unix/mpconfigport.mk` file, which has inline descriptions of the `ports/unix/mpconfigport.mk` file, which has inline descriptions of the
options. For example, to build SSL module (required for the `upip` tool options. For example, to build the SSL module, `MICROPY_PY_USSL` should be
described above, and so enabled by default), `MICROPY_PY_USSL` should be set set to 1.
to 1.

3
ports/unix/variants/manifest.py
View File

@ -1,2 +1 @@
module("upip.py", base_path="$(MPY_DIR)/tools", opt=3) require("mip")
module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)

351
tools/upip.py
View File

@ -1,351 +0,0 @@
#
# upip - Package manager for MicroPython
#
# Copyright (c) 2015-2018 Paul Sokolovsky
#
# Licensed under the MIT license.
#
import sys
import gc
import uos as os
import uerrno as errno
import ujson as json
import uzlib
import upip_utarfile as tarfile
gc.collect()
debug = False
index_urls = ["https://micropython.org/pi", "https://pypi.org/pypi"]
install_path = None
cleanup_files = []
gzdict_sz = 16 + 15
file_buf = bytearray(512)
class NotFoundError(Exception):
pass
def op_split(path):
if path == "":
return ("", "")
r = path.rsplit("/", 1)
if len(r) == 1:
return ("", path)
head = r[0]
if not head:
head = "/"
return (head, r[1])
# Expects *file* name
def _makedirs(name, mode=0o777):
ret = False
s = ""
comps = name.rstrip("/").split("/")[:-1]
if comps[0] == "":
s = "/"
for c in comps:
if s and s[-1] != "/":
s += "/"
s += c
try:
os.mkdir(s)
ret = True
except OSError as e:
if e.errno != errno.EEXIST and e.errno != errno.EISDIR:
raise e
ret = False
return ret
def save_file(fname, subf):
global file_buf
with open(fname, "wb") as outf:
while True:
sz = subf.readinto(file_buf)
if not sz:
break
outf.write(file_buf, sz)
def install_tar(f, prefix):
meta = {}
for info in f:
# print(info)
fname = info.name
try:
fname = fname[fname.index("/") + 1 :]
except ValueError:
fname = ""
save = True
for p in ("setup.", "PKG-INFO", "README"):
# print(fname, p)
if fname.startswith(p) or ".egg-info" in fname:
if fname.endswith("/requires.txt"):
meta["deps"] = f.extractfile(info).read()
save = False
if debug:
print("Skipping", fname)
break
if save:
outfname = prefix + fname
if info.type != tarfile.DIRTYPE:
if debug:
print("Extracting " + outfname)
_makedirs(outfname)
subf = f.extractfile(info)
save_file(outfname, subf)
return meta
def expandhome(s):
if "~/" in s:
h = os.getenv("HOME")
s = s.replace("~/", h + "/")
return s
import ussl
import usocket
warn_ussl = True
def url_open(url):
global warn_ussl
if debug:
print(url)
proto, _, host, urlpath = url.split("/", 3)
try:
port = 443
if ":" in host:
host, port = host.split(":")
port = int(port)
ai = usocket.getaddrinfo(host, port, 0, usocket.SOCK_STREAM)
except OSError as e:
fatal("Unable to resolve %s (no Internet?)" % host, e)
# print("Address infos:", ai)
ai = ai[0]
s = usocket.socket(ai[0], ai[1], ai[2])
try:
# print("Connect address:", addr)
s.connect(ai[-1])
if proto == "https:":
s = ussl.wrap_socket(s, server_hostname=host)
if warn_ussl:
print("Warning: %s SSL certificate is not validated" % host)
warn_ussl = False
# MicroPython rawsocket module supports file interface directly
s.write("GET /%s HTTP/1.0\r\nHost: %s:%s\r\n\r\n" % (urlpath, host, port))
l = s.readline()
protover, status, msg = l.split(None, 2)
if status != b"200":
if status == b"404" or status == b"301":
raise NotFoundError("Package not found")
raise ValueError(status)
while 1:
l = s.readline()
if not l:
raise ValueError("Unexpected EOF in HTTP headers")
if l == b"\r\n":
break
except Exception as e:
s.close()
raise e
return s
def get_pkg_metadata(name):
for url in index_urls:
try:
f = url_open("%s/%s/json" % (url, name))
except NotFoundError:
continue
try:
return json.load(f)
finally:
f.close()
raise NotFoundError("Package not found")
def fatal(msg, exc=None):
print("Error:", msg)
if exc and debug:
raise exc
sys.exit(1)
def install_pkg(pkg_spec, install_path):
package = pkg_spec.split("==")
data = get_pkg_metadata(package[0])
if len(package) == 1:
latest_ver = data["info"]["version"]
else:
latest_ver = package[1]
packages = data["releases"][latest_ver]
del data
gc.collect()
assert len(packages) == 1
package_url = packages[0]["url"]
print("Installing %s %s from %s" % (pkg_spec, latest_ver, package_url))
f1 = url_open(package_url)
try:
f2 = uzlib.DecompIO(f1, gzdict_sz)
f3 = tarfile.TarFile(fileobj=f2)
meta = install_tar(f3, install_path)
finally:
f1.close()
del f3
del f2
gc.collect()
return meta
def install(to_install, install_path=None):
# Calculate gzip dictionary size to use
global gzdict_sz
sz = gc.mem_free() + gc.mem_alloc()
if sz <= 65536:
gzdict_sz = 16 + 12
if install_path is None:
install_path = get_install_path()
if install_path[-1] != "/":
install_path += "/"
if not isinstance(to_install, list):
to_install = [to_install]
print("Installing to: " + install_path)
# sets would be perfect here, but don't depend on them
installed = []
try:
while to_install:
if debug:
print("Queue:", to_install)
pkg_spec = to_install.pop(0)
if pkg_spec in installed:
continue
meta = install_pkg(pkg_spec, install_path)
installed.append(pkg_spec)
if debug:
print(meta)
deps = meta.get("deps", "").rstrip()
if deps:
deps = deps.decode("utf-8").split("\n")
to_install.extend(deps)
except Exception as e:
print(
"Error installing '{}': {}, packages may be partially installed".format(pkg_spec, e),
file=sys.stderr,
)
def get_install_path():
global install_path
if install_path is None:
# sys.path[0] is current module's path
install_path = sys.path[1]
if install_path == ".frozen":
install_path = sys.path[2]
install_path = expandhome(install_path)
return install_path
def cleanup():
for fname in cleanup_files:
try:
os.unlink(fname)
except OSError:
print("Warning: Cannot delete " + fname)
def help():
print(
"""\
upip - Simple PyPI package manager for MicroPython
Usage: micropython -m upip install [-p <path>] <package>... | -r <requirements.txt>
import upip; upip.install(package_or_list, [<path>])
If <path> isn't given, packages will be installed to sys.path[1], or
sys.path[2] if the former is .frozen (path can be set from MICROPYPATH
environment variable if supported)."""
)
print("Default install path:", get_install_path())
print(
"""\
Note: only MicroPython packages (usually, named micropython-*) are supported
for installation, upip does not support arbitrary code in setup.py.
"""
)
def main():
global debug
global index_urls
global install_path
install_path = None
if len(sys.argv) < 2 or sys.argv[1] == "-h" or sys.argv[1] == "--help":
help()
return
if sys.argv[1] != "install":
fatal("Only 'install' command supported")
to_install = []
i = 2
while i < len(sys.argv) and sys.argv[i][0] == "-":
opt = sys.argv[i]
i += 1
if opt == "-h" or opt == "--help":
help()
return
elif opt == "-p":
install_path = sys.argv[i]
i += 1
elif opt == "-r":
list_file = sys.argv[i]
i += 1
with open(list_file) as f:
while True:
l = f.readline()
if not l:
break
if l[0] == "#":
continue
to_install.append(l.rstrip())
elif opt == "-i":
index_urls = [sys.argv[i]]
i += 1
elif opt == "--debug":
debug = True
else:
fatal("Unknown/unsupported option: " + opt)
to_install.extend(sys.argv[i:])
if not to_install:
help()
return
install(to_install)
if not debug:
cleanup()
if __name__ == "__main__":
main()

95
tools/upip_utarfile.py
View File

@ -1,95 +0,0 @@
import uctypes
# http://www.gnu.org/software/tar/manual/html_node/Standard.html
TAR_HEADER = {
"name": (uctypes.ARRAY | 0, uctypes.UINT8 | 100),
"size": (uctypes.ARRAY | 124, uctypes.UINT8 | 11),
}
DIRTYPE = "dir"
REGTYPE = "file"
def roundup(val, align):
return (val + align - 1) & ~(align - 1)
class FileSection:
def __init__(self, f, content_len, aligned_len):
self.f = f
self.content_len = content_len
self.align = aligned_len - content_len
def read(self, sz=65536):
if self.content_len == 0:
return b""
if sz > self.content_len:
sz = self.content_len
data = self.f.read(sz)
sz = len(data)
self.content_len -= sz
return data
def readinto(self, buf):
if self.content_len == 0:
return 0
if len(buf) > self.content_len:
buf = memoryview(buf)[: self.content_len]
sz = self.f.readinto(buf)
self.content_len -= sz
return sz
def skip(self):
sz = self.content_len + self.align
if sz:
buf = bytearray(16)
while sz:
s = min(sz, 16)
self.f.readinto(buf, s)
sz -= s
class TarInfo:
def __str__(self):
return "TarInfo(%r, %s, %d)" % (self.name, self.type, self.size)
class TarFile:
def __init__(self, name=None, fileobj=None):
if fileobj:
self.f = fileobj
else:
self.f = open(name, "rb")
self.subf = None
def next(self):
if self.subf:
self.subf.skip()
buf = self.f.read(512)
if not buf:
return None
h = uctypes.struct(uctypes.addressof(buf), TAR_HEADER, uctypes.LITTLE_ENDIAN)
# Empty block means end of archive
if h.name[0] == 0:
return None
d = TarInfo()
d.name = str(h.name, "utf-8").rstrip("\0")
d.size = int(bytes(h.size), 8)
d.type = [REGTYPE, DIRTYPE][d.name[-1] == "/"]
self.subf = d.subf = FileSection(self.f, d.size, roundup(d.size, 512))
return d
def __iter__(self):
return self
def __next__(self):
v = self.next()
if v is None:
raise StopIteration
return v
def extractfile(self, tarinfo):
return tarinfo.subf