This new logic tracks when an unconditional jump/raise occurs in the
emitted code stream (bytecode or native machine code) and suppresses all
subsequent code, until a label is assigned. This eliminates a lot of
cases of dead code, with relatively simple logic.
This commit combined with the previous one (that removed the existing
dead-code finding logic) has the following code size change:
bare-arm: -16 -0.028%
minimal x86: -60 -0.036%
unix x64: -368 -0.070%
unix nanbox: -80 -0.017%
stm32: -204 -0.052% PYBV10
cc3200: +0 +0.000%
esp8266: -232 -0.033% GENERIC
esp32: -224 -0.015% GENERIC[incl -40(data)]
mimxrt: -192 -0.054% TEENSY40
renesas-ra: -200 -0.032% RA6M2_EK
nrf: +28 +0.015% pca10040
rp2: -256 -0.050% PICO
samd: -12 -0.009% ADAFRUIT_ITSYBITSY_M4_EXPRESS
Signed-off-by: Damien George <damien@micropython.org>
The search in these cases should include all finally handlers that are
after the current ip. If a handler starts at exactly ip then it is
considered "after" the ip. This can happen when END_FINALLY is followed
immediately by a finally handler (from a different finally).
Consider the function:
def f():
try:
return 0
finally:
print(1)
The current bytecode emitter generates the following code:
00 SETUP_FINALLY 5
02 LOAD_CONST_SMALL_INT 0
03 RETURN_VALUE
04 LOAD_CONST_NONE ****
05 LOAD_GLOBAL print
07 LOAD_CONST_SMALL_INT 1
08 CALL_FUNCTION n=1 nkw=0
10 POP_TOP
11 END_FINALLY
12 LOAD_CONST_NONE
13 RETURN_VALUE
The LOAD_CONST_NONE marked with **** is dead code because it follows a
RETURN_VALUE, and nothing jumps to this LOAD_CONST_NONE. If the emitter
could remove this this dead code it would produce:
00 SETUP_FINALLY 4
02 LOAD_CONST_SMALL_INT 0
03 RETURN_VALUE
04 LOAD_GLOBAL print
06 LOAD_CONST_SMALL_INT 1
07 CALL_FUNCTION n=1 nkw=0
09 POP_TOP
10 END_FINALLY
11 LOAD_CONST_NONE
12 RETURN_VALUE
In this case the finally block (which starts at offset 4) immediately
follows the RETURN_VALUE. When RETURN_VALUE executes ip will point to
offset 4 in the bytecode (because the dispatch of the opcode does *ip++)
and so the finally handler will only be found if a >= comparison is used.
It's a similar story for break/continue:
while True:
try:
break
finally:
print(1)
Although technically in this case the > comparison still works because the
extra byte from the UNWIND_JUMP (encoding the number of exception handlers
to unwind) doesn't have a *ip++ (just a *ip) so ip remains pointing within
the UNWIND_JUMP opcode, and not at the start of the following finally
handler. Nevertheless, the change is made to use >= for consistency with
the RETURN_VALUE change.
Signed-off-by: Damien George <damien@micropython.org>
Catch calls to legacy:
MP_REGISTER_MODULE(name, module, enable)
Emit a friendly error suggesting they be rewritten to:
MP_REGISTER_MODULE(name, module).
Signed-off-by: Phil Howard <phil@pimoroni.com>
This file is not executable so shouldn't have the shebang line. This line
can cause issues when building on Windows msvc when the PyPython variable
is set to something other than "python", because it reverts back to using
the shebang line.
The top comment is also changed to """ style which matches all other
preprocessing scripts in the py/ directory.
Signed-off-by: Damien George <damien@micropython.org>
Without this, newer versions of gcc (eg 11.2.0) used with -O2 can warn
about `q_ptr` being maybe uninitialized, because it doesn't know that there
is at least one qstr being written in to this (alloca'd) memory.
As part of this, change the type of `n` to `size_t` so the compiler knows
it's unsigned and can generate better code.
Code size change for this commit:
bare-arm: -28 -0.049%
minimal x86: -4 -0.002%
unix x64: +0 +0.000%
unix nanbox: -16 -0.003%
stm32: -24 -0.006% PYBV10
cc3200: -32 -0.017%
esp8266: +8 +0.001% GENERIC
esp32: -52 -0.003% GENERIC
nrf: -24 -0.013% pca10040
rp2: -32 -0.006% PICO
samd: -28 -0.020% ADAFRUIT_ITSYBITSY_M4_EXPRESS
Signed-off-by: Damien George <damien@micropython.org>
This was made redundant by f2040bfc7e, which
also did not update this function for the change to qstr-opcode encoding,
so it does not work correctly anyway.
Signed-off-by: Damien George <damien@micropython.org>
Support for architecture-specific qstr linking was removed in
d4d53e9e11, where native code was changed to
access qstr values via qstr_table. The only remaining use for the special
qstr link table in persistentcode.c is to support native module written in
C, linked via mpy_ld.py. But native modules can also use the standard
module-level qstr_table (and obj_table) which was introduced in the .mpy
file reworking in f2040bfc7e.
This commit removes the remaining native qstr liking support in
persistentcode.c's load_raw_code function, and adds two new relocation
options for constants.qstr_table and constants.obj_table. mpy_ld.py is
updated to use these relocations options instead of the native qstr link
table.
Signed-off-by: Damien George <damien@micropython.org>
It's no longer needed because this macro is now processed after
preprocessing the source code via cpp (in the qstr extraction stage), which
means unused MP_REGISTER_MODULE's are filtered out by the preprocessor.
Signed-off-by: Damien George <damien@micropython.org>
This cleans up the parsing of MP_REGISTER_MODULE() and generation of
genhdr/moduledefs.h so that it uses the same process as compressed error
string messages, using the output of qstr extraction.
This makes sure all MP_REGISTER_MODULE()'s that are part of the build are
correctly picked up. Previously the extraction would miss some (eg if you
had a mod.c file in the board directory for an stm32 board).
Build speed is more or less unchanged.
Thanks to @stinos for the ports/windows/msvc/genhdr.targets changes.
Signed-off-by: Damien George <damien@micropython.org>
This allows mpy-cross to dynamically select whether ARMv7-M float
instructions are supported in @micropython.asm_thumb functions.
Signed-off-by: Damien George <damien@micropython.org>
This follows on from a5324a1074 and allows
mpy-cross to dynamically select whether ARMv7-M instructions are supported
in @micropython.asm_thumb functions.
The config option MICROPY_EMIT_INLINE_THUMB_ARMV7M is no longer needed, it
is now controlled by MICROPY_EMIT_THUMB_ARMV7M.
Signed-off-by: Damien George <damien@micropython.org>
The following changes are made:
- If MICROPY_VFS is enabled then mp_vfs_import_stat and mp_vfs_open are
automatically used for mp_import_stat and mp_builtin_open respectively.
- If MICROPY_PY_IO is enabled then "open" is automatically included in the
set of builtins, and points to mp_builtin_open_obj.
This helps to clean up and simplify the most common port configuration.
Signed-off-by: Damien George <damien@micropython.org>
The examples/natmod features0 and features1 examples now build and run on
ARMv6-M platforms. More complicated examples are not yet supported because
the compiler emits references to built-in functions like __aeabi_uidiv.
Signed-off-by: Damien George <damien@micropython.org>
If __thumb2__ is defined by the compiler then .mpy files marked as ARMV6M
and above (up to ARMV7EMDP) are supported. If it's not defined then only
ARMV6M .mpy files are supported. This makes sure that on CPUs like
Cortex-M0+ (where __thumb2__ is not defined) only .mpy files marked as
ARMV6M can be imported.
Signed-off-by: Damien George <damien@micropython.org>
This commit adjusts the asm_thumb_xxx functions so they can be dynamically
configured to use ARMv7-M instructions or not. This is available when
MICROPY_DYNAMIC_COMPILER is enabled, and then controlled by the value of
mp_dynamic_compiler.native_arch.
If MICROPY_DYNAMIC_COMPILER is disabled the previous behaviour is retained:
the functions emit ARMv7-M instructions only if MICROPY_EMIT_THUMB_ARMV7M
is enabled.
Signed-off-by: Damien George <damien@micropython.org>
This eliminates the need to save and restore the exception unwind handler
pointer when calling nlr_push.
Signed-off-by: Damien George <damien@micropython.org>
For example, ussl can come from axtls or mbedtls. If neither are enabled
then don't try and set an empty definition twice, and only include it
once in MICROPY_REGISTERED_MODULES.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Now that constant tuples are supported in the parser, eg (1, True, "str"),
it's a small step to allow anything that is a constant to be used with the
pattern:
from micropython import const
X = const(obj)
This commit makes the required changes to allow the following types of
constants:
from micropython import const
_INT = const(123)
_FLOAT = const(1.2)
_COMPLEX = const(3.4j)
_STR = const("str")
_BYTES = const(b"bytes")
_TUPLE = const((_INT, _STR, _BYTES))
_TUPLE2 = const((None, False, True, ..., (), _TUPLE))
Prior to this, only integers could be used in const(...).
Signed-off-by: Damien George <damien@micropython.org>
The recent rework of bytecode made all constants global with respect to the
module (previously, each function had its own constant table). That means
the constant table for a module is shared among all functions/methods/etc
within the module.
This commit add support to the compiler to de-duplicate constants in this
module constant table. So if a constant is used more than once -- eg 1.0
or (None, None) -- then the same object is reused for all instances.
For example, if there is code like `print(1.0, 1.0)` then the parser will
create two independent constants 1.0 and 1.0. The compiler will then (with
this commit) notice they are the same and only put one of them in the
constant table. The bytecode will then reuse that constant twice in the
print expression. That allows the second 1.0 to be reclaimed by the GC,
also means the constant table has one less entry so saves a word.
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, all qstrs were required to be allocated (by calling
mp_emit_common_use_qstr) in the MP_PASS_SCOPE pass (the first one). But
this is an unnecessary restriction, which is lifted by this commit.
Lifting the restriction simplifies the compiler because it can allocate
qstrs in later passes.
This also generates better code, because in some cases (eg when a variable
is closed over) the scope of an identifier is not known until a bit later
and then the identifier no longer needs its qstr allocated in the global
table.
Code size is reduced for all ports with this commit.
Signed-off-by: Damien George <damien@micropython.org>
Some architectures (like esp32 xtensa) cannot read byte-wise from
executable memory. This means the prelude for native functions -- which is
usually located after the machine code for the native function -- must be
placed in separate memory that can be read byte-wise. Prior to this commit
this was achieved by enabling N_PRELUDE_AS_BYTES_OBJ for the emitter and
MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ for the runtime. The prelude was
then placed in a bytes object, pointed to by the module's constant table.
This behaviour is changed by this commit so that a pointer to the prelude
is stored either in mp_obj_fun_bc_t.child_table, or in
mp_obj_fun_bc_t.child_table[num_children] if num_children > 0. The reasons
for doing this are:
1. It decouples the native emitter from runtime requirements, the emitted
code no longer needs to know if the system it runs on can/can't read
byte-wise from executable memory.
2. It makes all ports have the same emitter behaviour, there is no longer
the N_PRELUDE_AS_BYTES_OBJ option.
3. The module's constant table is now used only for actual constants in the
Python code. This allows further optimisations to be done with the
constants (eg constant deduplication).
Code size change for those ports that enable the native emitter:
unix x64: +80 +0.015%
stm32: +24 +0.004% PYBV10
esp8266: +88 +0.013% GENERIC
esp32: -20 -0.002% GENERIC[incl -112(data)]
rp2: +32 +0.005% PICO
Signed-off-by: Damien George <damien@micropython.org>
mpy-cross will now generate native code based on the size of
mp_code_state_native_t, and the runtime will use this struct to calculate
the offset of the .state field. This makes native code generation and
execution (which rely on this struct) independent to the settings
MICROPY_STACKLESS and MICROPY_PY_SYS_SETTRACE, both of which change the
size of the mp_code_state_t struct.
Fixes issue #5059.
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, even with unicode disabled .py and .mpy files could
contain unicode characters, eg by entering them directly in a string as
utf-8 encoded.
The only thing the compiler disallowed (with unicode disabled) was using
\uxxxx and \Uxxxxxxxx notation to specify a character within a string with
value >= 0x100; that would give a SyntaxError.
With this change mpy-cross will now accept \u and \U notation to insert a
character with value >= 0x100 into a string (because the -mno-unicode
option is now gone, there's no way to forbid this). The runtime will
happily work with strings with such characters, just like it already works
with strings with characters that were utf-8 encoded directly.
This change simplifies things because there are no longer any feature
flags in .mpy files, and any bytecode .mpy will now run on any target.
Signed-off-by: Damien George <damien@micropython.org>
This replaces occurences of
foo_t *foo = m_new_obj(foo_t);
foo->base.type = &foo_type;
with
foo_t *foo = mp_obj_malloc(foo_t, &foo_type);
Excludes any places where base is a sub-field or when new0/memset is used.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is to replace the following:
mp_foo_obj_t *self = m_new_obj(mp_foo_obj_t);
self->base.type = &mp_type_foo;
with:
mp_foo_obj_t *self = mp_obj_malloc(mp_foo_obj_t, &mp_type_foo);
Calling the function is less code than inlining setting the type
everywhere, adds up to ~100 bytes on PYBV11.
It also helps to avoid an easy mistake of forgetting to set the type.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When in a class body or at the module level don't implicitly close over
variables that have been assigned to.
Fixes issue #8603.
Signed-off-by: Damien George <damien@micropython.org>
This contains a string useful for identifying the underlying machine. This
string is kept consistent with the second part of the REPL banner via the
new config option MICROPY_BANNER_MACHINE.
This makes os.uname() more or less redundant, as all the information in
os.uname() is now available in the sys module.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds the git hash and build date to sys.version. This is
allowed according to CPython docs, and is what PyPy does. The docs state:
A string containing the version number of the Python interpreter plus
additional information on the build number and compiler used.
Eg on CPython:
Python 3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0]'
and PyPy:
Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)
[PyPy 5.6.0 with GCC 6.2.0 20161109] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> sys.version
'2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)\n[PyPy 5.6.0 with GCC ...
With this commit on MicroPython we now have:
MicroPython v1.18-371-g9d08eb024 on 2022-04-28; linux [GCC 11.2.0] v...
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import sys
>>> sys.version
'3.4.0; MicroPython v1.18-371-g9d08eb024 on 2022-04-28'
Note that the start of the banner is the same as the end of sys.version.
This helps to keep code size under control because the string can be reused
by the compiler.
Signed-off-by: Damien George <damien@micropython.org>
Entering tab at the REPL will now make it insert an indent (4 spaces) in
the following cases:
- after any whitespace on a line
- at the start of a line that is not the first line
This changes the existing behaviour where a tab would insert an indent only
if there were no matches in the auto-complete search, and it was the start
of the line. This means, if there were any symbols in the global
namespace, tab could never be used to indent.
Note that entering tab at the start of the first line will still do
auto-completion, but will now do nothing if there are no symbols in the
global namespace, which is more consistent than before.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds support to the parser so that tuples which contain only
constant elements (bool, int, str, bytes, etc) are immediately converted to
a tuple object. This makes it more efficient to use tuples containing
constant data because they no longer need to be created at runtime by the
bytecode (or native code).
Furthermore, with this improvement constant tuples that are part of frozen
code are now able to be stored fully in ROM (this will be implemented in
later commits).
Code size is increased by about 400 bytes on Cortex-M4 platforms.
See related issue #722.
Signed-off-by: Damien George <damien@micropython.org>
To keep the separate parts of the code that use these values in sync. And
make it easier to add new object types.
Signed-off-by: Damien George <damien@micropython.org>
This reverts commit 7e8222ae06.
The prelude data must exist somewhere in the native code so load_raw_code
and mpy-tool.py can access and parse it.
Signed-off-by: Damien George <damien@micropython.org>
If MICROPY_SCHEDULER_STATIC_NODES is enabled then C code can declare a
static mp_sched_node_t and schedule a callback using
mp_sched_schedule_node(). In contrast to using mp_sched_schedule(), the
node version will have at most one pending callback outstanding, and will
always be able to schedule if there is nothing already scheduled on this
node. This guarantees that the the callback will be called exactly once
after it is scheduled.
Signed-off-by: Damien George <damien@micropython.org>
The Thumb instruction set has special 16 bit encodings for PUSH involving
LR and POP involving PC, which are commonly used in nested functions.
Using this encoding is particularly important for ARMv6-M, where the more
general 32 bit encoding of PUSH and POP is unavailable.
- The classification of source files in makeqstrdefs.py has been moved into
functions to consolidate the logic for that classification into a single
place.
- Classification of source files (into C or C++ or "other" files) is based
on the filename extension.
- For C++ there are many more common filename extensions than just ".cpp";
see "Options Controlling the Kind of Output" in man gcc for example. All
common extensions for C++ source files which need preprocessing have been
added.
The values are always real objects, only the key can be MP_OBJ_NULL to
indicate a **kwargs entry.
Signed-off-by: Damien George <damien@micropython.org>
There were two issues with the existing code:
1. "1 << i" is computed as a 32-bit number so would overflow when
executed on 64-bit machines (when mp_uint_t is 64-bit). This meant that
*args beyond 32 positions would not be handled correctly.
2. star_args must fit as a positive small int so that it is encoded
correctly in the emitted code. MP_SMALL_INT_BITS is too big because it
overflows a small int by 1 bit. MP_SMALL_INT_BITS - 1 does not work
because it produces a signed small int which is then sign extended when
extracted (even by mp_obj_get_int_truncated), and this sign extension
means that any position arg after *args is also treated as a star-arg.
So the maximum bit position is MP_SMALL_INT_BITS - 2. This means that
MP_OBJ_SMALL_INT_VALUE() can be used instead of
mp_obj_get_int_truncated() to get the value of star_args.
These issues are fixed by this commit, and a test added.
Signed-off-by: Damien George <damien@micropython.org>
This replaces instances of uint with size_t and int with ssize_t in
the mp_call_prepare_args_n_kw_var() function since all of the variables
are used as array offsets.
Also sort headers while we are touching this.
Signed-off-by: David Lechner <david@pybricks.com>
To reach this check, n_kw has to be >= 1 and therefore args2_alloc has
to be >= 2. Therefore new_alloc will always be >= 4. So this check will
never be true and can be removed.
Signed-off-by: David Lechner <david@pybricks.com>
This fixes overallocating an extra mp_obj_t when the length of *args and
**args is known. Previously we were allocating 1 mp_obj_t for each
n_args and n_kw plus the length of each *arg and **arg (if they are
known). Since n_args includes *args and n_kw includes **args, this was
allocating an extra mp_obj_t in addition to the length of these args
when unpacked.
To fix this, we just subtract 1 from the length to account for the 1
already implicitly allocated by n_args and n_kw.
Signed-off-by: David Lechner <david@pybricks.com>
This is a partial implementation of PEP 448 to allow unpacking multiple
star args in a function or method call.
This is implemented by changing the emitted bytecodes so that both
positional args and star args are stored as positional args. A bitmap is
added to indicate if an argument at a given position is a positional
argument or a star arg.
In the generated code, this new bitmap takes the place of the old star arg.
It is stored as a small int, so this means only the first N arguments can
be star args where N is the number of bits in a small int.
The runtime is modified to interpret this new bytecode format while still
trying to perform as few memory reallocations as possible.
Signed-off-by: David Lechner <david@pybricks.com>
This is a partial implementation of PEP 448 to allow multiple ** unpackings
when calling a function or method.
The compiler is modified to encode the argument as a None: obj key-value
pair (similar to how regular keyword arguments are encoded as str: obj
pairs). The extra object that was pushed on the stack to hold a single **
unpacking object is no longer used and is removed.
The runtime is modified to decode this new format.
Signed-off-by: David Lechner <david@pybricks.com>
This warning can happen on clang 13.0.1 building mpy-cross:
../py/vm.c:748:25: error: array index -3 refers past the last possible
element for an array in 64-bit address space containing 64-bit (8-byte)
elements (max possible 2305843009213693952 elements)
[-Werror,-Warray-bounds]
sp[-MP_OBJ_ITER_BUF_NSLOTS + 1] = MP_OBJ_NULL;
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
Using pointer access instead of array access works around this warning.
Fixes issue #8467.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds optimised l32i/s32i functions that select the best load/
store encoding based on the size of the offset, and uses the function when
necessary in code generation.
Without this, ASM_LOAD_REG_REG_OFFSET() could overflow the word offset
(using a narrow encoding), for example when loading the prelude from the
constant table when there are many (>16) constants.
Fixes issue #8458.
Signed-off-by: Damien George <damien@micropython.org>
The sys module should always be available (if it's compiled in), eg to
change sys.path for importing. So provide an explicit alias from "sys" to
"usys" so that "import sys" can always work.
Signed-off-by: Damien George <damien@micropython.org>
These jumps are always forwards, and it's more efficient in the VM to
decode an unsigned argument. These opcodes are already optimised versions
of the sequence "dup-top pop-jump-if-x pop" so it doesn't hurt generality
to optimise them further.
Signed-off-by: Damien George <damien@micropython.org>
This commit introduces changes:
- All jump opcodes are changed to have variable length arguments, of either
1 or 2 bytes (previously they were fixed at 2 bytes). In most cases only
1 byte is needed to encode the short jump offset, saving bytecode size.
- The bytecode emitter now selects 1 byte jump arguments when the jump
offset is guaranteed to fit in 1 byte. This is achieved by checking if
the code size changed during the last pass and, if it did (if it shrank),
then requesting that the compiler make another pass to get the correct
offsets of the now-smaller code. This can continue multiple times until
the code stabilises. The code can only ever shrink so this iteration is
guaranteed to complete. In most cases no extra passes are needed, the
original 4 passes are enough to get it right by the 4th pass (because the
2nd pass computes roughly the correct labels and the 3rd pass computes
the correct size for the jump argument).
This change to the jump opcode encoding reduces .mpy files and RAM usage
(when bytecode is in RAM) by about 2% on average.
The performance of the VM is not impacted, at least within measurment of
the performance benchmark suite.
Code size is reduced for builds that include a decent amount of frozen
bytecode. ARM Cortex-M builds without any frozen code increase by about
350 bytes.
Signed-off-by: Damien George <damien@micropython.org>
Some compilers will warn about unused variables like scope_flags. So use
MP_BC_PRELUDE_SIG_DECODE() which will silence these warnings.
Signed-off-by: Damien George <damien@micropython.org>
This adds a new MP_SMALL_INT_BITS macro that is a compile-time constant
that contains the number of bits available in an MP_SMALL_INT.
We can use this in place of the runtime function mp_small_int_bits().
Signed-off-by: David Lechner <david@pybricks.com>
The bytecode state variables mp_showbc_code_start and mp_showbc_constants
have been removed and made local variables passed into the various
functions.
As part of this, the DECODE_PTR macro is fixed so it extracts the relevant
pointer from the child_table (a regression introduced in
f2040bfc7e).
Signed-off-by: Damien George <damien@micropython.org>
This means that all constants for EMIT_ARG(load_const_obj, obj) are created
in the parser (rather than some in the compiler).
Signed-off-by: Damien George <damien@micropython.org>
This commit adds generic support for mutable module attributes on built in
modules, by adding support for an optional hook function for module
attribute lookup. If a module wants to support additional attribute load/
store/delete (beyond what is in the constant, globals dict) then it should
add at the very end of its globals dict MP_MODULE_ATTR_DELEGATION_ENTRY().
This should point to a custom function which will handle any additional
attributes.
The mp_module_generic_attr() function is provided as a helper function for
additional attributes: it requires an array of qstrs (terminated in
MP_QSTRnull) and a corresponding array of objects (with a 1-1 mapping
between qstrs and objects). If the qstr is found in the array then the
corresponding object is loaded/stored/deleted.
Signed-off-by: Damien George <damien@micropython.org>
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing. They are also
smaller on disk.
But the real benefit of .mpy files comes when they are frozen into the
firmware. This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device. These C data structures can be executed in-place, ie directly from
ROM. This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).
The downside of frozen code is that it requires recompiling and reflashing
the entire firmware. This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).
This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware. The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place. If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.
With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).
The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded. Instead only a small qstr table needs to be built (and put in RAM)
at import time. This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory. Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).
In more detail, in the VM what used to be (schematically):
qst = DECODE_QSTR_VALUE;
is now (schematically):
idx = DECODE_QSTR_INDEX;
qst = qstr_table[idx];
That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values. Only qstr_table needs to be linked
when the .mpy is loaded.
Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.
The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before
The qstr indirection in the bytecode has only a small impact on VM
performance. For stm32 on PYBv1.0 the performance change of this commit
is:
diff of scores (higher is better)
N=100 M=100 baseline -> this-commit diff diff% (error%)
bm_chaos.py 371.07 -> 357.39 : -13.68 = -3.687% (+/-0.02%)
bm_fannkuch.py 78.72 -> 77.49 : -1.23 = -1.563% (+/-0.01%)
bm_fft.py 2591.73 -> 2539.28 : -52.45 = -2.024% (+/-0.00%)
bm_float.py 6034.93 -> 5908.30 : -126.63 = -2.098% (+/-0.01%)
bm_hexiom.py 48.96 -> 47.93 : -1.03 = -2.104% (+/-0.00%)
bm_nqueens.py 4510.63 -> 4459.94 : -50.69 = -1.124% (+/-0.00%)
bm_pidigits.py 650.28 -> 644.96 : -5.32 = -0.818% (+/-0.23%)
core_import_mpy_multi.py 564.77 -> 581.49 : +16.72 = +2.960% (+/-0.01%)
core_import_mpy_single.py 68.67 -> 67.16 : -1.51 = -2.199% (+/-0.01%)
core_qstr.py 64.16 -> 64.12 : -0.04 = -0.062% (+/-0.00%)
core_yield_from.py 362.58 -> 354.50 : -8.08 = -2.228% (+/-0.00%)
misc_aes.py 429.69 -> 405.59 : -24.10 = -5.609% (+/-0.01%)
misc_mandel.py 3485.13 -> 3416.51 : -68.62 = -1.969% (+/-0.00%)
misc_pystone.py 2496.53 -> 2405.56 : -90.97 = -3.644% (+/-0.01%)
misc_raytrace.py 381.47 -> 374.01 : -7.46 = -1.956% (+/-0.01%)
viper_call0.py 576.73 -> 572.49 : -4.24 = -0.735% (+/-0.04%)
viper_call1a.py 550.37 -> 546.21 : -4.16 = -0.756% (+/-0.09%)
viper_call1b.py 438.23 -> 435.68 : -2.55 = -0.582% (+/-0.06%)
viper_call1c.py 442.84 -> 440.04 : -2.80 = -0.632% (+/-0.08%)
viper_call2a.py 536.31 -> 532.35 : -3.96 = -0.738% (+/-0.06%)
viper_call2b.py 382.34 -> 377.07 : -5.27 = -1.378% (+/-0.03%)
And for unix on x64:
diff of scores (higher is better)
N=2000 M=2000 baseline -> this-commit diff diff% (error%)
bm_chaos.py 13594.20 -> 13073.84 : -520.36 = -3.828% (+/-5.44%)
bm_fannkuch.py 60.63 -> 59.58 : -1.05 = -1.732% (+/-3.01%)
bm_fft.py 112009.15 -> 111603.32 : -405.83 = -0.362% (+/-4.03%)
bm_float.py 246202.55 -> 247923.81 : +1721.26 = +0.699% (+/-2.79%)
bm_hexiom.py 615.65 -> 617.21 : +1.56 = +0.253% (+/-1.64%)
bm_nqueens.py 215807.95 -> 215600.96 : -206.99 = -0.096% (+/-3.52%)
bm_pidigits.py 8246.74 -> 8422.82 : +176.08 = +2.135% (+/-3.64%)
misc_aes.py 16133.00 -> 16452.74 : +319.74 = +1.982% (+/-1.50%)
misc_mandel.py 128146.69 -> 130796.43 : +2649.74 = +2.068% (+/-3.18%)
misc_pystone.py 83811.49 -> 83124.85 : -686.64 = -0.819% (+/-1.03%)
misc_raytrace.py 21688.02 -> 21385.10 : -302.92 = -1.397% (+/-3.20%)
The code size change is (firmware with a lot of frozen code benefits the
most):
bare-arm: +396 +0.697%
minimal x86: +1595 +0.979% [incl +32(data)]
unix x64: +2408 +0.470% [incl +800(data)]
unix nanbox: +1396 +0.309% [incl -96(data)]
stm32: -1256 -0.318% PYBV10
cc3200: +288 +0.157%
esp8266: -260 -0.037% GENERIC
esp32: -216 -0.014% GENERIC[incl -1072(data)]
nrf: +116 +0.067% pca10040
rp2: -664 -0.135% PICO
samd: +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS
As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.
In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place. Performance is not impacted too much. Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM. This will
essentially be able to replace frozen code for most applications.
Signed-off-by: Damien George <damien@micropython.org>
This allows the compiler to merge strings: e.g. "update",
"difference_update" and "symmetric_difference_update" will all point to the
same memory.
No functional change.
The size reduction depends on the number of qstrs in the build. The change
this commit brings is:
bare-arm: -4 -0.007%
minimal x86: +150 +0.092% [incl +48(data)]
unix x64: -608 -0.118%
unix nanbox: -572 -0.126% [incl +32(data)]
stm32: -1392 -0.352% PYBV10
cc3200: -448 -0.244%
esp8266: -1208 -0.173% GENERIC
esp32: -1028 -0.068% GENERIC[incl -1020(data)]
nrf: -440 -0.252% pca10040
rp2: -1072 -0.217% PICO
samd: -368 -0.264% ADAFRUIT_ITSYBITSY_M4_EXPRESS
Performance is also improved (on bare metal at least) for the
core_import_mpy_multi.py, core_import_mpy_single.py and core_qstr.py
performance benchmarks.
Originally at adafruit#4583
Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
The inclusion of `umachine` in the list of built-in modules is now done
centrally in py/objmodule.c. Enabling MICROPY_PY_MACHINE will include this
module.
As part of this, all ports now have `umachine` as the core module name
(previously some had only `machine` as the name).
Signed-off-by: Damien George <damien@micropython.org>
The new test has an .exp file, because it is not compatible with Python 3.9
and lower.
See CPython version of the issue at https://bugs.python.org/issue27772
Signed-off-by: Jeff Epler <jepler@gmail.com>
The qstr_last_chunk is not collected by the garbage collector. This relies
on the assertion that qstr_pool_t also references the qstr_last_chunk. If
an exception is raised while allocating the qstr_pool_t, qstr_last_chunk
has to be invalidated not to become a dangling reference at the next
garbage collection.
Signed-off-by: Emilie Feral <emilie.feral@numworks.com>
In commit 86ce442607 the '.frozen' entry was
added at the start of sys.path, to allow control over when frozen modules
are searched during import, and retain existing behaviour whereby frozen
was searched before the filesystem.
But Python semantics of sys.path require sys.path[0] to be the directory of
the currently executing script, or ''.
This commit moves the '.frozen' entry to second place in sys.path, so
sys.path[0] retains its correct value (described above).
Signed-off-by: Damien George <damien@micropython.org>
This commit makes sure that the value zero is always encoded in an mpz_t as
neg=0 and len=0 (previously it was just len=0).
This invariant is needed for some of the bitwise operations that operate on
negative numbers, because they cannot handle -0. For example
(-((1<<100)-(1<<100)))|1 was being computed as -65535, instead of 1.
Fixes issue #8042.
Signed-off-by: Damien George <damien@micropython.org>
The mp_sys_path_obj and mp_sys_argv_obj objects are only used by the
runtime and accessible from Python if MICROPY_PY_SYS is enabled. So
exclude them from the runtime state if this option is disabled.
Signed-off-by: Damien George <damien@micropython.org>
If MICROPY_PY_SYS_PATH_ARGV_DEFAULTS is enabled (which it is by default)
then sys.path and sys.argv will be initialised and populated with default
values. This keeps all bare-metal ports aligned.
Signed-off-by: Damien George <damien@micropython.org>
This ensures MICROPY_QSTR_EXTRA_POOL and MICROPY_MODULE_FROZEN_MPY are set
if necessary before the CFLAGS are extracted for QSTR generation.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This changes makemanifest.py & mpy-tool.py to merge string and mpy names
into the same list (now mp_frozen_names).
The various paths for loading a frozen module (mp_find_frozen_module) and
checking existence of a frozen module (mp_frozen_stat) use a common
function that searches this list.
In addition, the frozen lookup will now only take place if the path starts
with ".frozen", which needs to be added to sys.path.
This fixes issues #1804, #2322, #3509, #6419.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This feature is not enabled on any port, it's not in CPython's io module,
and functionality is better suited to the micropython-lib implementation of
pkg_resources.
Make it possible to proceed to a regular lookup in locals_dict if the
custom type->attr fails. This allows type->attr to extend rather than
completely replace the lookup in locals_dict.
This is useful for custom builtin classes that have mostly regular methods
but just a few special attributes/properties. This way, type->attr needs
to deal with the special cases only and the default lookup will be used for
generic methods.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
This makes it possible for cooperative multitasking systems to keep running
event loops during garbage collector operations.
For example, this can be used to ensure that a motor control loop runs
approximately each 5 ms. Without this hook, the loop time can jump to
about 15 ms.
Addresses #3475.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
This is an stm32-specific feature that's accessed via the pyb module, so
not something that will be widely enabled.
Signed-off-by: Damien George <damien@micropython.org>
Some of these will later be moved to CORE or BASIC, but EXTRA is a good
starting point based on what stm32 uses.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.
This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.
The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).
It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.
For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:
diff of scores (higher is better)
N=2000 M=2000 bccache -> attrmapcache diff diff% (error%)
bm_chaos.py 13742.56 -> 13905.67 : +163.11 = +1.187% (+/-3.75%)
bm_fannkuch.py 60.13 -> 61.34 : +1.21 = +2.012% (+/-2.11%)
bm_fft.py 113083.20 -> 114793.68 : +1710.48 = +1.513% (+/-1.57%)
bm_float.py 256552.80 -> 243908.29 : -12644.51 = -4.929% (+/-1.90%)
bm_hexiom.py 521.93 -> 625.41 : +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py 197544.25 -> 217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py 8072.98 -> 8198.75 : +125.77 = +1.558% (+/-3.22%)
misc_aes.py 17283.45 -> 16480.52 : -802.93 = -4.646% (+/-0.82%)
misc_mandel.py 99083.99 -> 128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py 83860.10 -> 82592.56 : -1267.54 = -1.511% (+/-2.27%)
misc_raytrace.py 21490.40 -> 22227.23 : +736.83 = +3.429% (+/-1.88%)
This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).
The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):
diff of scores (higher is better)
N=2000 M=2000 native -> nat-attrmapcache diff diff% (error%)
bm_chaos.py 14130.62 -> 15464.68 : +1334.06 = +9.441% (+/-7.11%)
bm_fannkuch.py 74.96 -> 76.16 : +1.20 = +1.601% (+/-1.80%)
bm_fft.py 166682.99 -> 168221.86 : +1538.87 = +0.923% (+/-4.20%)
bm_float.py 233415.23 -> 265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py 628.59 -> 734.17 : +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py 225418.44 -> 232926.45 : +7508.01 = +3.331% (+/-3.10%)
bm_pidigits.py 6322.00 -> 6379.52 : +57.52 = +0.910% (+/-5.62%)
misc_aes.py 20670.10 -> 27223.18 : +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py 138221.11 -> 152014.01 : +13792.90 = +9.979% (+/-2.46%)
misc_pystone.py 85032.14 -> 105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py 19800.01 -> 23350.73 : +3550.72 = +17.933% (+/-2.79%)
In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.
See #7680 for further discussion. And see also #7653 for a discussion
about simplifying mpy-cross options.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The existing inline bytecode caching optimisation, selected by
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, reserves an extra byte in the
bytecode after certain opcodes, which at runtime stores a map index of the
likely location of this field when looking up the qstr. This scheme is
incompatible with bytecode-in-ROM, and doesn't work with native generated
code. It also stores bytecode in .mpy files which is of a different format
to when the feature is disabled, making generation of .mpy files more
complex.
This commit provides an alternative optimisation via an approach that adds
a global cache for map offsets, then all mp_map_lookup operations use it.
It's less precise than bytecode caching, but allows the cache to be
independent and external to the bytecode that is executing. It also works
for the native emitter and adds a similar performance boost on top of the
gain already provided by the native emitter.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When the LOAD_ATTR opcode is executed there are quite a few different cases
that have to be handled, but the common case is accessing a member on an
instance type. Typically, built-in types provide methods which is why this
is common.
Fortunately, for this specific case, if the member is found in the member
map then there's no further processing.
This optimisation does a relatively cheap check (type is instance) and then
forwards directly to the member map lookup, falling back to the regular
path if necessary.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is the beginning of a set of changes to simplify enabling/disabling
features. The goals are:
- Remove redundancy from mpconfigport.h (never set a value to the default
-- make it clear exactly what's being enabled).
- Improve consistency between ports. All "similar" ports (i.e. approx same
flash size) should get the same features.
- Simplify mpconfigport.h -- just get default/sensible options for the size
of the port.
- Make it easy for defining constrained boards (e.g. STM32F0/L0), they can
just set a lower level.
This commit makes a step towards this and defines the "core" level as the
current default feature set, and a "minimal" level to turn off everything.
And a few placeholder levels are added for where the other ports will
roughly land.
This is a no-op change for all ports.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit simplifies and optimises the parse tree in-memory
representation of lists of expressions, for tuples and lists, and when
tuples are used on the left-hand-side of assignments and within del
statements. This reduces memory usage of the parse tree when such code is
compiled, and also reduces the size of the compiler.
For example, (1,) was previously the following parse tree:
expr_stmt(5) (n=2)
atom_paren(45) (n=1)
testlist_comp(146) (n=2)
int(1)
testlist_comp_3b(149) (n=1)
NULL
NULL
and with this commit is now:
expr_stmt(5) (n=2)
atom_paren(45) (n=1)
testlist_comp(146) (n=1)
int(1)
NULL
Similarly, (1, 2, 3) was previously:
expr_stmt(5) (n=2)
atom_paren(45) (n=1)
testlist_comp(146) (n=2)
int(1)
testlist_comp_3c(150) (n=2)
int(2)
int(3)
NULL
and is now:
expr_stmt(5) (n=2)
atom_paren(45) (n=1)
testlist_comp(146) (n=3)
int(1)
int(2)
int(3)
NULL
Signed-off-by: Damien George <damien@micropython.org>
This commit refactors machine.PWM and creates extmod/machine_pwm.c. The
esp8266, esp32 and rp2 ports all use this and provide implementations of
the required PWM functionality. This helps to reduce code duplication and
keep the same Python API across ports.
This commit does not make any functional changes.
Signed-off-by: Damien George <damien@micropython.org>
The zephyr port doesn't support SoftI2C so it's not enabled, and the legacy
I2C constructor check can be removed.
Signed-off-by: Damien George <damien@micropython.org>
This is a generic API for synchronously bit-banging data on a pin.
Initially this adds a single supported encoding, which supports controlling
WS2812 LEDs.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This was missed in 692d36d779. It's not
strictly necessary as the GC will clean it anyway, but it's good to
pre-emptively gc_free() all the blocks used in lexing/parsing.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.
It is implemented in the lexer as a syntax translation to `str.format`:
f"{a}" --> "{}".format(a)
It also supports:
f"{a=}" --> "a={}".format(a)
This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.
There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
`SyntaxError: raw f-strings are not supported`.
- literal concatenation of f-strings with adjacent strings will fail
"{}" f"{a}" --> "{}{}".format(a) (str.format will incorrectly use
the braces from the non-f-string)
f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)
- PEP-498 requires the full parser to understand the interpolated
argument, however because this entirely runs in the lexer it cannot
resolve nested braces in expressions like
f"{'}'}"
- The !r, !s, and !a conversions are not supported.
Includes tests and cpydiffs.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Commit 4173950658 removed automatic building
of mpy-cross, which rebuilt it whenever any of its dependent source files
changed.
But needing to build mpy-cross, and not knowing how, is a frequent issue.
This commit aims to help by automatically building mpy-cross only if it
doesn't exist. For Makefiles it uses an order-only prerequisite, while
for CMake it uses a custom command.
If MICROPY_MPYCROSS (which is what makemanifest.py uses to locate the
mpy-cross executable) is defined in the environment then automatic build
will not be attempted, allowing a way to prevent this auto-build if needed.
Thanks to Trammell Hudson aka @osresearch for the original idea; see #5760.
Signed-off-by: Damien George <damien@micropython.org>
Optionally enabled via MICROPY_PY_UJSON_SEPARATORS. Enabled by default.
For dump, make sure mp_get_stream_raise is called after
mod_ujson_separators since CPython does it in this order (if both
separators and stream are invalid, separators will raise an exception
first).
Add separators argument in the docs as well.
Signed-off-by: Peter Züger <zueger.peter@icloud.com>
Signed-off-by: Damien George <damien@micropython.org>
Damien George
Phil Howard
Jeremy Herbert
Andrew Leech
Jim Mussared
Jon Bjarni Bjarnason
Christian Zietz
Daniel Jour
David Lechner
David Lechner
stijn
Artyom Skrobov
Jeff Epler
Emilie Feral
Laurens Valk
Mike Wadsten
iabdalkader
Peter Züger