Commit Graph

4250 Commits

Author SHA1 Message Date
Jim Mussared 6873a14b61 py/makeversionhdr.py: Allow running outside of repo.
If a CMake-build is run with `make BUILD=/outside/path` then
makeversionheader.py is run with the CWD set to the build directory, which
means the git version lookup will fail and silently fall back to the
mpconfig.h mode (giving the wrong result).

This commit:
 - Uses the location of makeversionheader.py to find the repo path.
 - Allows overriding this path via --repo-path.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-11-22 14:36:21 +11:00
Jan Willeke 40a3aa709c py/nlrmips: Add native NLR support for MIPS architecture.
This can be tested using ports/minimal and qemu:

    make CC=mips-linux-gnu-gcc-8

Then run with qemu-mips:

    stty raw opost -echo;
    QEMU_LD_PREFIX=/usr/mips-linux-gnu/ qemu-mips build/firmware.elf;
    sleep 1; reset

Signed-off-by: Jan Willeke <willeke@smartmote.de>
2022-11-15 17:09:49 +11:00
Damien George 0698dd72ea py/emitnative: Ensure load_subscr does not clobber existing REG_ARG_2.
Follow up from a similar fix in 426785a19e

Fixes issue #6314.

Signed-off-by: Damien George <damien@micropython.org>
2022-11-11 12:25:32 +11:00
Jim Mussared 2c8dab7ab4 py/objarray: Detect bytearray(str) without an encoding.
This prevents a very subtle bug caused by writing e.g. `bytearray('\xfd')`
which gives you `(0xc3, 0xbd)`.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-11-08 23:09:22 +11:00
Simon Arlott f8b0ae32d3 py/builtinimport: Fix crash handling "weak link" module names.
There are two calls to mp_builtin___import__():
1. ports/unix/main.c:main_() which provides a str in args[0]
2. py/runtime.c:mp_import_name() which provides a qstr in args[0]

The default implementation of mp_builtin___import__() is
mp_builtin___import___default() which has a different implementation based
on MICROPY_ENABLE_EXTERNAL_IMPORT.

If MICROPY_ENABLE_EXTERNAL_IMPORT is disabled then the handling of weak
links assumes that args[0] is a `const char *`, when it is either a str or
qstr object.

Use the existing qstr of the module name instead, and also use a vstr
instead of strcpy() to ensure no overflow occurs.
2022-11-08 23:03:21 +11:00
Damien George cd35b8a2a3 extmod/machine_timer: Move stm32's implementation of machine.Timer here.
So it can be reused by other ports.

Signed-off-by: Damien George <damien@micropython.org>
2022-10-27 14:32:43 +11:00
Damien George c138e10fbb py/makeversionhdr: Fall back to py/mpconfig.h instead of docs/conf.py.
Commit 64af916c11 removed the version string
from docs/conf.py.  py/mpconfig.h is a better place to get the version
from, so use that (when there is no git repository).

Signed-off-by: Damien George <damien@micropython.org>
2022-10-27 13:00:48 +11:00
Jim Mussared d75c7e822c py/obj: Add comments explaining the slot index scheme.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-26 00:08:41 +11:00
Jim Mussared 1ba0e8ff96 py/persistentcode: Only emit sub-version if generated code has native.
In order for v1.19.1 to load a .mpy, the formerly-feature-flags which are
now used for the sub-version must be zero.

The sub-version is only used to indicate a native version change, so it
should be zero when emitting bytecode-only .mpy files.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-25 14:57:04 +11:00
Jim Mussared b161abc574 py/obj: Verify floating point type is correct for repr C.
Prevents double-precision floats being enabled on 32-bit architectures
where they will not fit into the mp_obj_t encoding.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-25 14:46:04 +11:00
David Lechner ab317a0d66 py/misc: Remove use of bitfield from vstr_t.
Since there is only one flag, we don't need to use a bitfield in vstr_t.
Compilers emit extra instructions to access a bitfield, so this should
reduce the binary size a small amount.

Signed-off-by: David Lechner <david@pybricks.com>
2022-10-12 00:33:09 +11:00
Jim Mussared d6d8722558 extmod: Make extmod.mk self-contained.
This makes it so that all a port needs to do is set the relevant variables
and "include extmod.mk" and doesn't need to worry about adding anything to
OBJ, CFLAGS, SRC_QSTR, etc.

Make all extmod variables (src, flags, etc) private to extmod.mk.

Also move common/shared, extmod-related fragments (e.g. wiznet, cyw43,
bluetooth) into extmod.mk.

Now that SRC_MOD, CFLAGS_MOD, CXXFLAGS_MOD are unused by both extmod.mk
(and user-C-modules in a previous commit), remove all uses of them from
port makefiles.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 23:31:49 +11:00
Jim Mussared 3cc6decfc4 py/py.mk: Make user-C-module handling self-contained in py.mk.
Removes the need for the port to add anything to OBJS or SRC_QSTR.

Also makes it possible for user-C-modules to differentiate between code
that should be processed for QSTR vs other files (e.g. helpers and
libraries).

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 23:23:18 +11:00
Jim Mussared 0e8dfaf538 py/modsys: Add support for sys.executable.
Only intended to be used on Unix and other "OS" ports.  Matches CPython.
This should give the absolute path to the executing binary.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-10-11 18:10:26 +11:00
Jim Mussared c44b3927b8 py/objstr: Add a helper to set mp_obj_str_t data.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 17:50:19 +11:00
stijn fecfbc3f67 py/mkenv.mk: Make CPP definition explicit for consistency. 2022-09-29 14:17:37 +02:00
Matt Trentini 0bc1d10557 rp2/Makefile: Add support for BOARD_VARIANTS.
Following stm32.  This allows a single board definition to define variants
of its configuration.
2022-09-26 12:38:58 +10:00
Jim Mussared 9d6f474ea4 py/objstr: Don't treat bytes as unicode in str.count.
`b'\xaa \xaa'.count(b'\xaa')` now (correctly) returns 2 instead of 1.

Fixes issue #9404.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-26 00:54:18 +10:00
Angus Gratton 25ff5b52d9 py/parse: Allow const types other than int to optimise as true/false.
Allows optimisation of cases like:

    import micropython
    _DEBUG = micropython.const(False)
    if _DEBUG:
        print('Debugging info')

Previously the 'if' statement was only optimised out if the type of the
const() argument was integer.

The change is implemented in a way that makes the compiler slightly smaller
(-16 bytes on PYBV11) but compilation will also be very slightly slower.

As a bonus, if const support is enabled then the compiler can now optimise
const truthy/falsey expressions of other types, like:

    while "something":
        pass

... unclear if that is useful, but perhaps it could be.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2022-09-23 16:04:13 +10:00
Damien George fb77be1506 py: Include filename in errors from loading/saving files via "open".
This improves error messages in mpy-cross:
- When loading a .py file that doesn't exist (or can't be opened) it now
  includes the filename in the OSError.
- When saving a .mpy file that can't be opened it now raises an exception
  (prior, it would silently fail), and includes the filename in the
  OSError.

Signed-off-by: Damien George <damien@micropython.org>
2022-09-19 23:27:08 +10:00
Damien George cc588ac3a9 py/runtime: Add mp_raise_OSError_with_filename helper function.
Useful when more detail is needed for an OSError associated with a file.

Signed-off-by: Damien George <damien@micropython.org>
2022-09-19 23:25:26 +10:00
Jim Mussared 15d0615d5c py/objmodule: Add support for __dict__.
This matches class `__dict__`, and is similarly gated on
MICROPY_CPYTHON_COMPAT. Unlike class though, because modules's globals are
actually dict instances, the result is a mutable dictionary.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 23:22:46 +10:00
Jim Mussared d94141e147 py/persistentcode: Introduce .mpy sub-version.
The intent is to allow us to make breaking changes to the native ABI (e.g.
changes to dynruntime.h) without needing the bytecode version to increment.

With this commit the two bits previously used for the feature flags (but
now unused as of .mpy version 6) encode a sub-version.  A bytecode-only
.mpy file can be loaded as long as MPY_VERSION matches, but a native .mpy
(i.e. one with an arch set) must also match MPY_SUB_VERSION.  This allows 3
additional updates to the native ABI per bytecode revision.

The sub-version is set to 1 because the previous commits that changed the
layout of mp_obj_type_t have changed the native ABI.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-09-19 23:19:55 +10:00
Jim Mussared b41aaaa8a9 py/obj: Optimise code size and performance for make_new as a slot.
The check for make_new (i.e. used to determine something's type) is now
more complicated due to the slot access.  This commit changes the inlining
of a few frequently-used helpers to overall improve code size and
performance.
2022-09-19 19:06:16 +10:00
Jim Mussared 94beeabd2e py/obj: Convert make_new into a mp_obj_type_t slot.
Instead of being an explicit field, it's now a slot like all the other
methods.

This is a marginal code size improvement because most types have a make_new
(100/138 on PYBV11), however it improves consistency in how types are
declared, removing the special case for make_new.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:15 +10:00
Jim Mussared 6da41b5900 py/obj: Merge getiter and iternext mp_obj_type_t slots.
The goal here is to remove a slot (making way to turn make_new into a slot)
as well as reduce code size by the ~40 references to mp_identity_getiter
and mp_stream_unbuffered_iter.

This introduces two new type flags:
- MP_TYPE_FLAG_ITER_IS_ITERNEXT: This means that the "iter" slot in the
  type is "iternext", and should use the identity getiter.
- MP_TYPE_FLAG_ITER_IS_CUSTOM: This means that the "iter" slot is a pointer
  to a mp_getiter_iternext_custom_t instance, which then defines both
  getiter and iternext.

And a third flag that is the OR of both, MP_TYPE_FLAG_ITER_IS_STREAM: This
means that the type should use the identity getiter, and
mp_stream_unbuffered_iter as iternext.

Finally, MP_TYPE_FLAG_ITER_IS_GETITER is defined as a no-op flag to give
the default case where "iter" is "getiter".

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:13 +10:00
Jim Mussared 3c6127dfcf py/objnamedtuple: Optimise slot RAM usage for namedtuple.
Rather than reserving a full 12-slot mp_obj_type_t, reserve enough room for
seven and cast as necessary.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:13 +10:00
Jim Mussared 165388e4eb py/objtype: Optimise slot RAM usage for instance types.
In all cases other than where you have a native base with a protocol, it
now fits into 4 GC blocks (like it did before the slots representation).

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:13 +10:00
Jim Mussared cb0ffdd2bf py/obj: Remove basic mp_obj_type_t sparse representation.
This makes the slots-based representation the only option.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:12 +10:00
Jim Mussared 3ac8b5851e py/obj: Add slot-index mp_obj_type_t representation.
The existings mp_obj_type_t uses a sparse representation for slots for the
capability methods of the type (eg print, make_new).  This commit adds a
compact slot-index representation.  The basic idea is that where the
mp_obj_type_t struct used to have 12 pointer fields, it now has 12 uint8_t
indices, and a variable-length array of pointers.  So in the best case (no
fields used) it saves 12x4-12=36 bytes (on a 32-bit machine) and in the
common case (three fields used) it saves 9x4-12=24 bytes.

Overall with all associated changes, this slot-index representation reduces
code size by 1000 to 3000 bytes on bare-metal ports.  Performance is
marginally better on a few tests (eg about 1% better on misc_pystone.py and
misc_raytrace.py on PYBv1.1), but overall marginally worse by a percent or
so.

See issue #7542 for further analysis and discussion.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:09 +10:00
Jim Mussared a52cd5b07d py/obj: Add accessors for type slots and use everywhere.
This is a no-op, but sets the stage for changing the mp_obj_type_t
representation.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:07 +10:00
Jim Mussared e8355eb163 py/obj: Add "full" and "empty" non-variable-length mp_obj_type_t.
This will always have the maximum/minimum size of a mp_obj_type_t
representation and can be used as a member in other structs.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:04 +10:00
Jim Mussared 5ddf671944 py/objexcept: Make MP_DEFINE_EXCEPTION use MP_DEFINE_CONST_OBJ_TYPE.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:01 +10:00
Jim Mussared 9dce82776d all: Remove unnecessary locals_dict cast.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:01 +10:00
Jim Mussared b7d6ee9b75 all: Fix #if inside MP_DEFINE_CONST_OBJ_TYPE for msvc.
Changes:

    MP_DEFINE_CONST_OBJ_TYPE(
       ...
       #if FOO
       ...
       #endif
       ...
    );

to:

    MP_DEFINE_CONST_OBJ_TYPE(
       ...
       FOO_TYPE_ATTR
       ...
    );

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:01 +10:00
Jim Mussared 662b9761b3 all: Make all mp_obj_type_t defs use MP_DEFINE_CONST_OBJ_TYPE.
In preparation for upcoming rework of mp_obj_type_t layout.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:01 +10:00
Jim Mussared cdb880789f py/obj: Add macro to declare ROM mp_obj_type_t instances.
This will allow the structure of mp_obj_type_t to change while keeping the
definition code the same.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:05:50 +10:00
Jim Mussared fb2a57800a all: Simplify buffer protocol to just a "get buffer" callback.
The buffer protocol type only has a single member, and this existing layout
creates problems for the upcoming split/slot-index mp_obj_type_t layout
optimisations.

If we need to make the buffer protocol more sophisticated in the future
either we can rely on the mp_obj_type_t optimisations to just add
additional slots to mp_obj_type_t or re-visit the buffer protocol then.

This change is a no-op in terms of generated code.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 18:40:39 +10:00
Jim Mussared 45972fa548 py/mkrules.mk: Add link to build troubleshooting on failure.
Also update the submodules help text to match.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-15 12:22:18 +10:00
Damien George 74805435f9 py/objpolyiter: Fix comment about finaliser method.
Signed-off-by: Damien George <damien@micropython.org>
2022-09-13 21:14:22 +10:00
Jim Mussared 3e5b1be8ca py/mpconfig: Add "everything" features from unix coverage.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-13 17:40:33 +10:00
Jim Mussared 605266ee9a py/mpconfig: Make feature levels available to mpconfigport.h.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-13 17:37:49 +10:00
Jim Mussared 89a0fefb6c py/mpconfig: Add LFS1/LFS2 options to match FAT/posix.
Also fixes the #ifndef for FAT & posix.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-13 17:37:32 +10:00
Andrew Leech 582b3e4e78 py/objpolyiter: Add a new polyiter type with finaliser support. 2022-09-13 13:00:25 +10:00
Andrew Leech d521899e18 py/persistentcode: Clarify ValueError when native emitter disabled. 2022-08-29 12:38:49 +10:00
Jim Mussared 6c3d8d38bf py/objstr: Always validate utf-8 for mp_obj_new_str.
All uses of this are either tiny strings or not-known-to-be-safe.

Update comments for mp_obj_new_str_copy and mp_obj_new_str_of_type.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 16:45:46 +10:00
Jim Mussared 3a910b1565 py/objstr: Optimise mp_obj_new_str_from_vstr for known-safe strings.
The new `mp_obj_new_str_from_utf8_vstr` can be used when you know you
already have a unicode-safe string.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 16:44:35 +10:00
Jim Mussared 88864587f5 py/objstr: Always ensure mp_obj_str_from_vstr is unicode-safe.
Now that we have `mp_obj_new_str_type_from_vstr` (private helper used by
objstr.c) split from the public API (`mp_obj_new_str_from_vstr`), we can
enforce a unicode check at the public API without incurring a performance
cost on the various objstr.c methods (which are already working on known
unicode-safe strings).

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 16:44:20 +10:00
Jim Mussared 8a0ee5a5c0 py/objstr: Split mp_obj_str_from_vstr into bytes/str versions.
Previously the desired output type was specified.  Now make the type part
of the function name.  Because this function is used in a few places this
saves code size due to smaller call-site.

This makes `mp_obj_new_str_type_from_vstr` a private function of objstr.c
(which is almost the only place where the output type isn't a compile-time
constant).

This saves ~140 bytes on PYBV11.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 16:43:55 +10:00
Laurens Valk d8ad87843a py/builtinimport: Allow overriding of mp_builtin___import__.
This allows ports to override mp_builtin___import__.

This can be useful in MicroPython applications where
MICROPY_ENABLE_EXTERNAL_IMPORT has to be disabled due to its impact on
build size (2% to 2.5% of the minimal port). By overriding the otherwise
very minimal mp_builtin___import__, ports can still allow limited forms
of application-specific imports.

Signed-off-by: Laurens Valk <laurens@pybricks.com>
2022-08-23 13:34:06 +10:00
Damien George 3d65101a8a py: Clean up formatting of union definitions.
Signed-off-by: Damien George <damien@micropython.org>
2022-08-23 13:09:57 +10:00
Jim Mussared af1f167820 py/dynruntime: Add mp_obj_is_true.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-19 23:31:11 +10:00
Damien George 8f4c108025 all: Remove MICROPY_PY_IO_FILEIO config option.
Since commit e65d1e69e8 there is no longer an
io.FileIO class, so this option is no longer needed.

This option also controlled whether or not files supported being opened in
binary mode (eg 'rb'), and could, if disabled, lead to confusion as to why
opening a file in binary mode silently did the wrong thing (it would just
open in text mode if MICROPY_PY_IO_FILEIO was disabled).

The various VFS implementations (POSIX, FAT, LFS) were the only places
where enabling this option made a difference, and in almost all cases where
one of these filesystems were enabled, MICROPY_PY_IO_FILEIO was also
enabled.  So it makes sense to just unconditionally enable this feature
(ability to open a file in binary mode) in all cases, and so just remove
this config option altogether.  That makes configuration simpler and means
binary file support always exists (and opening a file in binary mode is
arguably more fundamental than opening in text mode, so if anything should
be configurable then it should be the ability to open in text mode).

Signed-off-by: Damien George <damien@micropython.org>
2022-08-18 11:54:17 +10:00
Dan Ellis 6f4d424f46 py/formatfloat: Use pow(10, e) instead of pos/neg_pow lookup tables.
Rework the conversion of floats to decimal strings so it aligns precisely
with the conversion of strings to floats in parsenum.c.  This is to avoid
rendering 1eX as 9.99999eX-1 etc.  This is achieved by removing the power-
of-10 tables and using pow() to compute the exponent directly, and that's
done efficiently by first estimating the power-of-10 exponent from the
power-of-2 exponent in the floating-point representation.

Code size is reduced by roughly 100 to 200 bytes by this commit.

Signed-off-by: Dan Ellis <dan.ellis@gmail.com>
2022-08-12 23:53:34 +10:00
Dan Ellis 6cd2e41918 py/parsenum: Ensure that trailing zeros lead to identical results.
Prior to this commit, parsenum would calculate "1e-20" as 1.0*pow(10, -20),
and "1.000e-20" as 1000.0*pow(10, -23); in certain cases, this could make
seemingly-identical values compare as not equal.  This commit watches for
trailing zeros as a special case, and ignores them when appropriate, so
"1.000e-20" is also calculated as 1.0*pow(10, -20).

Fixes issue #5831.
2022-08-12 23:44:11 +10:00
Damien George cf90e24335 py/mkrules: Use abspath to find directory for mpy-cross dependency.
Otherwise if the `mpy-cross/build/` directory doesn't exist then
`mpy-cross/build/..` won't work.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-12 16:38:24 +10:00
Damien George 945f377b43 py/objstr: Remove str function object declarations from header file.
Since f7f56d4285 consolidated all uses of
these to a single locals dict, they no longer need to be made public.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-12 16:38:22 +10:00
Jim Mussared 28aaab9590 py/objstr: Add hex/fromhex to bytes/memoryview/bytearray.
These were added in Python 3.5.

Enabled via MICROPY_PY_BUILTINS_BYTES_HEX, and enabled by default for all
ports that currently have ubinascii.

Rework ubinascii to use the implementation of these methods.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-12 12:44:30 +10:00
Andrew Leech f7f56d4285 py/objstr: Consolidate methods for str/bytes/bytearray/array.
This commit adds the bytes methods to bytearray, matching CPython.  The
existing implementations of these methods for str/bytes are reused for
bytearray with minor updates to match CPython return types.

For details on the CPython behaviour see
https://docs.python.org/3/library/stdtypes.html#bytes-and-bytearray-operations

The work to merge locals tables for str/bytes/bytearray/array was done by
@jimmo.  Because of this merging of locals the change in code size for this
commit is mostly negative:

       bare-arm:    +0 +0.000%
    minimal x86:   +29 +0.018%
       unix x64:  -792 -0.128% standard[incl -448(data)]
    unix nanbox:  -436 -0.078% nanbox[incl -448(data)]
          stm32:   -40 -0.010% PYBV10
         cc3200:   -32 -0.017%
        esp8266:   -28 -0.004% GENERIC
          esp32:   -72 -0.005% GENERIC[incl -200(data)]
         mimxrt:   -40 -0.011% TEENSY40
     renesas-ra:   -40 -0.006% RA6M2_EK
            nrf:   -16 -0.009% pca10040
            rp2:   -64 -0.013% PICO
           samd:  +148 +0.105% ADAFRUIT_ITSYBITSY_M4_EXPRESS
2022-08-11 23:18:02 +10:00
Damien George 82b3500724 py/qstr: Change qstr hash type from mp_uint_t to size_t.
The hash is either 8 or 16 bits (depending on MICROPY_QSTR_BYTES_IN_HASH)
so will fit in a size_t.

This saves 268 bytes on the unix nanbox build.  Non-nanbox configurations
are unchanged because mp_uint_t is the same size as size_t.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-11 23:18:02 +10:00
Efi Weiss f3285fef07 py/nlrpowerpc: Fix generation of ppc64 code on ppc32 build.
Due to inline assembly, wrong instructions were generated.  Use
corresponding 32 bit instructions and fix the offsets used.

Signed-off-by: Efi Weiss <efiwiss@gmail.com>
2022-08-11 14:04:13 +10:00
Mat Booth 2e8816de91 py/dynruntime.mk: Allow building assembly source in natmods.
Allow inclusion of assembly source files in dynamic native modules.
2022-08-11 14:00:13 +10:00
Daniel Jour 47c84286e8 all: Fix paths to mpy-cross and micropython binaries.
Binaries built using the Make build system now no longer appear in the
working directory of the build, but rather in the build directory.  Thus
some paths had to be adjusted.
2022-08-11 13:31:13 +10:00
Daniel Jour b2e8240268 py/mkrules.mk: Keep all build artefacts inside $(BUILD) directory.
The rules for lib (static library with name $(LIBMICROPYTHON)) and the
default rule to build a binary (name $(PROG)) produced outputs in the
current working directory.  Change this to build these files in the build
directory.

Note: An empty BUILD variable can cause issues (references to the root
directory); this is not addressed by this commit due to multiple other
places having the same issue.
2022-08-11 13:29:44 +10:00
Damien George b5986784e4 py/objstr: Reformat str access macros to make them readable.
Signed-off-by: Damien George <damien@micropython.org>
2022-08-10 14:31:06 +10:00
Damien George 7d91a9bf5b py/mpprint: Fix formatting typo with mp_print_ext_t struct name.
Signed-off-by: Damien George <damien@micropython.org>
2022-08-10 14:30:47 +10:00
David Lechner 6baeded322 py/runtime: Fix crash in star arg unpacking.
The reallocation trigger for unpacking star args with unknown length
did not take into account the number of fixed args remaining. So it was
possible that the unpacked iterators could take up exactly the memory
allocated then nothing would be left for fixed args after the star args.
This causes a segfault crash.

This is fixed by taking into account the remaining number of fixed args
in the check to decide whether to realloc yet or not.

Signed-off-by: David Lechner <david@pybricks.com>
2022-08-06 11:32:58 -05:00
Jim Mussared 579f330508 py/mkenv.mk: Use micropython-lib from submodule by default.
Also adds micropython-lib to 'make submodules' when using a frozen manifest
(for make and cmake).

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-03 00:08:41 +10:00
Angus Gratton 1230d86dca py/builtinimport: Remove duplicate static function argument.
context==mc in all cases where this function was being called.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2022-07-27 11:38:47 +10:00
Dan Ellis f9cbe6bc47 py/formatfloat: Format all whole-number floats exactly.
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place.  This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point.  The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.

Fixes issue #4212.

Signed-off-by: Dan Ellis dan.ellis@gmail.com
2022-07-26 22:23:47 +10:00
Jim Mussared e65d1e69e8 py/modio: Remove FileIO and TextIOWrapper from io module.
On ports with more than one filesystem, the type will be wrong, for example
if using LFS but FAT enabled, then the type will be FAT.  So it's not
possible to use these classes to identify a file object type.

Furthermore, constructing an io.FileIO currently crashes on FAT, and
make_new isn't supported on LFS.

And the io.TextIOWrapper class does not match CPython at all.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-07-26 17:58:01 +10:00
Damien George c0fa903d6b py/compile: Support large integers in inline-asm data directive.
Fixes issue #8956.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-26 12:24:50 +10:00
Damien George 4fe3e493b1 py/obj: Make mp_obj_get_complex_maybe call mp_obj_get_float_maybe first.
This commit simplifies mp_obj_get_complex_maybe() by first calling
mp_obj_get_float_maybe() to handle the cases corresponding to floats.
Only if that fails does it attempt to extra a full complex number.

This reduces code size and also means that mp_obj_get_complex_maybe() now
supports user-defined classes defining __float__; in particular this allows
user-defined classes to be used as arguments to cmath-module function.

Furthermore, complex_make_new() can now be simplified to directly call
mp_obj_get_complex(), instead of mp_obj_get_complex_maybe() followed by
mp_obj_get_float().  This also improves error messages from complex with
an invalid argument, it now raises "can't convert <type> to complex" rather
than "can't convert <type> to float".

Signed-off-by: Damien George <damien@micropython.org>
2022-07-25 16:11:26 +10:00
Andrew Leech 1e87b56219 py/obj: Add support for __float__ and __complex__ functions. 2022-07-25 14:23:34 +10:00
Rob Knegjens 4a48531803 py/gc: Reduce code size when MICROPY_GC_SPLIT_HEAP is disabled.
Use C macros to reduce the size of firmware images when the GC split-heap
feature is disabled.

The code size difference of this commit versus HEAD~2 (ie the commit prior
to MICROPY_GC_SPLIT_HEAP being introduced) when split-heap is disabled is:

       bare-arm:    +0 +0.000%
    minimal x86:    +0 +0.000%
       unix x64:   -16 -0.003%
    unix nanbox:   -20 -0.004%
          stm32:    -8 -0.002% PYBV10
         cc3200:    +0 +0.000%
        esp8266:    +8 +0.001% GENERIC
          esp32:    +0 +0.000% GENERIC
            nrf:   -20 -0.011% pca10040
            rp2:    +0 +0.000% PICO
           samd:    -4 -0.003% ADAFRUIT_ITSYBITSY_M4_EXPRESS

The code size difference of this commit versus HEAD~2 split-heap is enabled
with MICROPY_GC_MULTIHEAP=1 (but no extra code to add more heaps):

    unix x64: +1032 +0.197% [incl +544(bss)]
       esp32:  +592 +0.039% GENERIC[incl +16(data) +264(bss)]
2022-07-23 00:43:08 +10:00
Ayke van Laethem bcc827d695 py/gc: Allow the GC heap to be split over multiple memory areas.
This commit adds a new option MICROPY_GC_SPLIT_HEAP (disabled by default)
which, when enabled, allows the GC heap to be split over multiple memory
areas/regions.  The first area is added with gc_init() and subsequent areas
can be added with gc_add().  New areas can be added at runtime.  Areas are
stored internally as a linked list, and calls to gc_alloc() can be
satisfied from any area.

This feature has the following use-cases (among others):
- The ESP32 has a fragmented OS heap, so to use all (or more) of it the
  GC heap must be split.
- Other MCUs may have disjoint RAM regions and are now able to use them
  all for the GC heap.
- The user could explicitly increase the size of the GC heap.
- Support a dynamic heap while running on an OS, adding more heap when
  necessary.
2022-07-23 00:42:54 +10:00
stijn e82aa2abc4 py/qstr: Make mp_decompress_rom_string decl and def the same.
Fixes MSVC warning about mismatching argument types.
2022-07-18 23:27:28 +10:00
stijn 1f16d682da py/misc: Fix msvc compilation with compressed error messages. 2022-07-18 23:25:12 +10:00
David Lechner a1ef5ac65d py/scheduler: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register sched_queue
instead of using a conditional inside of mp_state_vm_t.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:52:01 +10:00
David Lechner 85b4f36100 py/modsys: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register cur_exception,
sys_exitfunc, mp_sys_path_obj, mp_sys_argv_obj and sys_mutable
instead of using a conditional inside of mp_state_vm_t.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:52:01 +10:00
David Lechner a98aa66df6 py/persistentcode: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register track_reloc_code_list
instead of using a conditional inside of mp_state_vm_t.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:52:01 +10:00
David Lechner 2c728c5330 extmod/modbluetooth: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register `bluetooth`
instead of using a conditional inside of mp_state_vm_t.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:52:01 +10:00
David Lechner 32e32bd761 extmod/vfs: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register vfs_cur and
vfs_mount_table instead of using a conditional inside of mp_state_vm_t.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:52:01 +10:00
David Lechner d532c55e3b extmod/modlwip: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register lwip_slip_stream
instead of using a conditional inside of mp_state_vm_t.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:52:01 +10:00
David Lechner 631b692177 extmod/uos_dupterm: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register dupterm_objs
instead of using a conditional inside of mp_state_vm_t.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:52:01 +10:00
David Lechner 68f46342aa shared/runtime/pyexec: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register repl_line
instead of using a conditional inside of mp_state_vm_t.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:52:01 +10:00
David Lechner 7e4b205cb0 py/mpstate: Drop MICROPY_PORT_ROOT_POINTERS from mp_state_vm_t.
All in-tree uses of MICROPY_PORT_ROOT_POINTERS have been replaced with
MP_REGISTER_ROOT_POINTER(), so now we can remove both
MICROPY_PORT_ROOT_POINTERS and MICROPY_BOARD_ROOT_POINTERS from the code
and remaining config files.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:51:16 +10:00
David Lechner 81dbea1ce3 shared/readline: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register the readline_history root
pointer array used by shared/readline.c and removes the registration from
all mpconfigport.h files.

This also required adding a new MICROPY_READLINE_HISTORY_SIZE config option
since not all ports used the same sized array.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:48:49 +10:00
David Lechner fc3d7ae11b py/make_root_pointers: Add MP_REGISTER_ROOT_POINTER parser/generator.
This adds new compile-time infrastructure to parse source code files for
`MP_REGISTER_ROOT_POINTER()` and generates a new `root_pointers.h` header
file containing the collected declarations.  This works the same as the
existing `MP_REGISTER_MODULE()` feature.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:48:23 +10:00
Yonatan Goldschmidt a8d78cc398 py/obj: Add debug-only runtime checks to mp_obj_is_type().
Zero effect on non debug builds, and also usually optimized out even in
debug builds as mp_obj_is_type() is called with a compile-time known type.
I'm not sure we even have dynamic uses of mp_obj_is_type() at the moment,
but if we ever will they will be protected from now on.

Signed-off-by: Yonatan Goldschmidt <yon.goldschmidt@gmail.com>
2022-07-18 11:17:49 +10:00
Yonatan Goldschmidt 2a6ba47110 py/obj: Add static safety checks to mp_obj_is_type().
Commit d96cfd13e3 introduced a regression by breaking existing
users of mp_obj_is_type(.., &mp_obj_bool).  This function (and associated
helpers like mp_obj_is_int()) have some specific nuances, and mistakes like
this one can happen again.

This commit adds mp_obj_is_exact_type() which behaves like the the old
mp_obj_is_type().  The new mp_obj_is_type() has the same prototype but it
attempts to statically assert that it's not called with types which should
be checked using mp_obj_is_type().  If called with any of these types: int,
str, bool, NoneType - it will cause a compilation error.  Additional
checked types (e.g function types) can be added in the future.

Existing users of mp_obj_is_type() with the now "invalid" types, were
translated to use mp_obj_is_exact_type().

The use of MP_STATIC_ASSERT() is not bulletproof - usually GCC (and other
compilers) can't statically check conditions that are only known during
link-time (like variables' addresses comparison).  However, in this case,
GCC is able to statically detect these conditions, probably because it's
the exact same object - `&mp_type_int == &mp_type_int` is detected.
Misuses of this function with runtime-chosen types (e.g:
`mp_obj_type_t *x = ...; mp_obj_is_type(..., x);` won't be detected.  MSC
is unable to detect this, so we use MP_STATIC_ASSERT_NOT_MSC().

Compiling with this commit and without the fix for d96cfd13e3 shows
that it detects the problem.

Signed-off-by: Yonatan Goldschmidt <yon.goldschmidt@gmail.com>
2022-07-18 11:17:46 +10:00
Yonatan Goldschmidt 6670281472 py/misc: Add MP_STATIC_ASSERT_NOT_MSC().
To be used in cases where the condition of the assert does not compile
under msvc.

Signed-off-by: Yonatan Goldschmidt <yon.goldschmidt@gmail.com>
2022-07-18 11:11:00 +10:00
Lars Haulin 5bf3765631 py/objnamedtuple: Fix segfault with empty namedtuple.
The empty tuple is usually a constant object, but named tuples must be
allocated to allow modification.  Added explicit allocation to fix this.

Also added a regression test to verify creating an empty named tuple works.

Fixes issue #7870.

Signed-off-by: Lars Haulin <lars.haulin@gmail.com>
2022-07-13 16:25:35 +10:00
Damien George b878fc042f py/vm: Consistently indent #if guards to match the code they surround.
Signed-off-by: Damien George <damien@micropython.org>
2022-07-12 22:48:55 +10:00
Damien George 893a5c8341 py/vm: In YIELD_FROM opcode, expand helper macros and remove them.
The GENERATOR_EXIT_IF_NEEDED macro is only used once and it's easier to
read and understand the code if this macro body is written in the code.
Then the comment just before it makes more sense.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-12 22:48:07 +10:00
Damien George d84220b8c6 py/vm: Remove check for ip being NULL when handling StopIteration.
This check for code_state->ip being NULL was added in
a7c02c4538 with a commit message that "When
generator raises exception, it is automatically terminated (by setting its
code_state.ip to 0)".  It was also added without any tests to test for this
particular case.  (The commit did mention that CPython's test_pep380.py
triggered a bug, but upon re-running this test it did not show any need for
this NULL check of code_state->ip.)

It is true that generators that have completed (either by running to their
end or raising an exception) set "code_state.ip = 0".  But there is an
explicit check at the start of mp_obj_gen_resume() to return immediately
for any attempt to resume an already-stopped generator.  So the VM can
never execute a generator with NULL ip (and this was true at the time of
the above-referenced commit).

Furthermore, the other parts of the VM just before and after this piece
of code do require (or at least assume) code_state->ip is non-NULL.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-12 18:17:44 +10:00
Jim Mussared 9714a0ead5 py/emitnative: Fix STORE_ATTR viper code-gen when value is not a pyobj.
There was a missing call to MP_F_CONVERT_NATIVE_TO_OBJ.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-07-12 17:18:27 +10:00
Jim Mussared 158f1794e8 py/vm: Document internal SELECTIVE_EXC_IP option.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-07-12 16:13:14 +10:00
Jim Mussared 8db99f11a7 py/scheduler: De-inline and fix race with pending exception / scheduler.
The optimisation that allows a single check in the VM for either a pending
exception or non-empty scheduler queue doesn't work when threading is
enabled, as one thread can clear the sched_state if it has no pending
exception, meaning the thread with the pending exception will never see it.

This removes that optimisation for threaded builds.

Also fixes a race in non-scheduler builds where get-and-clear of the
pending exception is not protected by the atomic section.

Also removes the bulk of the inlining of pending exceptions and scheduler
handling from the VM. This just costs code size and complexity at no
performance benefit.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-07-12 15:54:33 +10:00
Damien George f1b5761ced py/mkrules.cmake: Improve printing of git-submodules error.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-30 13:49:47 +10:00
Damien George 6e83bb47eb py/builtinhelp: Don't show help for an MP_MODULE_ATTR_DELEGATION_ENTRY.
Otherwise it can lead to a crash.

Fixes issue #8816.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-28 16:35:01 +10:00
Damien George 5b66d08609 py/builtin: Remove unnecessary module declarations.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-27 22:17:13 +10:00
Damien George e22b7fb4af py/objfun: Support function attributes on native functions.
Native functions can just reuse the bytecode function attribute code.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-25 00:22:15 +10:00
Michael Bentley d68532558d py/objclosure: Forward function attributes for closures.
Add .attr attribute which forwards to self->fun.

A closure is intended to wrap around a function object, so forward any
requested attributes to the wrapped function object.

Signed-off-by: Michael Bentley <mikebentley15@gmail.com>
2022-06-24 23:46:59 +10:00
Damien George 627ba38154 py/parsenum: Optimise when building with complex disabled.
To reduce code size when MICROPY_PY_BUILTINS_COMPLEX is disabled.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-23 11:46:47 +10:00
Damien George 61ce260ff7 py/parsenum: Fix parsing of complex "j" and also "nanj", "infj".
Prior to this commit, complex("j") would return 0j, and complex("nanj")
would return nan+0j.  This commit makes sure "j" is tested for after
parsing the number (nan, inf or a decimal), and also supports the case of
"j" on its own.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-23 11:46:47 +10:00
Jim Mussared 0172292762 py/parsenum: Support parsing complex numbers of the form "a+bj".
To conform with CPython.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-06-23 11:46:47 +10:00
David Lechner c118b5d0e4 extmod/extmod.mk: Separate out extmod file list from py.mk to extmod.mk.
This separates extmod source files from `py.mk`.  Previously, `py.mk`
assumed that every consumer of the py/ directory also wanted to include
extmod/.  However, this is not the case.  For example, building mpy-cross
uses py/ but doesn't need extmod/.

This commit moves all extmod-specific items from `py.mk` to `extmod.mk` and
explicitly includes `extmod.mk` in ports that use it.

Signed-off-by: David Lechner <david@pybricks.com>
2022-06-21 00:14:34 +10:00
Damien George f5769698e5 extmod/modlwip: Clean up inclusion of modlwip in build process.
The following changes are made:

- Guard entire file with MICROPY_PY_LWIP, so it can be included in the
  build while still being disabled (for consistency with other extmod
  modules).

- Add modlwip.c to list of all extmod source in py/py.mk and
  extmod/extmod.cmake so all ports can easily use it.

- Move generic modlwip GIT_SUBMODULES build configuration code from
  ports/rp2/CMakeLists.txt to extmod/extmod.cmake, so it can be reused by
  other ports.

- Remove now unnecessary inclusion of modlwip.c in EXTMOD_SRC_C in esp8266
  port, and in SRC_QSTR in mimxrt port.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-20 23:37:38 +10:00
Damien George 5d3a0bb59c py/objcell: Make cell get/set funcs static-inline to reduce code size.
Change in code size is:

       bare-arm:   -36 -0.062%
    minimal x86:   -92 -0.056%
       unix x64:   -72 -0.014%
    unix nanbox:  -276 -0.060%
          stm32:    +0 +0.000% PYBV10
          stm32:   -40 +0.021% NUCLEO_L073RZ
         cc3200:   -16 -0.009%
        esp8266:  +176 +0.025% GENERIC
          esp32:   -28 -0.002% GENERIC
         mimxrt:   -56 -0.016% TEENSY40
     renesas-ra:    +0 +0.000% RA6M2_EK
            nrf:    +0 +0.000% pca10040
            rp2:   -64 -0.013% PICO
           samd:   -32 -0.023% ADAFRUIT_ITSYBITSY_M4_EXPRESS

Ports like stm32 that build the VM with -O3 have no change because the
savings from the inlining are offset by additional gcc performance
optimisations in the VM.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-20 23:37:38 +10:00
Damien George a506335524 py/emit: Suppress unreachable bytecode/native code that follows jump.
This new logic tracks when an unconditional jump/raise occurs in the
emitted code stream (bytecode or native machine code) and suppresses all
subsequent code, until a label is assigned.  This eliminates a lot of
cases of dead code, with relatively simple logic.

This commit combined with the previous one (that removed the existing
dead-code finding logic) has the following code size change:

       bare-arm:   -16 -0.028%
    minimal x86:   -60 -0.036%
       unix x64:  -368 -0.070%
    unix nanbox:   -80 -0.017%
          stm32:  -204 -0.052% PYBV10
         cc3200:    +0 +0.000%
        esp8266:  -232 -0.033% GENERIC
          esp32:  -224 -0.015% GENERIC[incl -40(data)]
         mimxrt:  -192 -0.054% TEENSY40
     renesas-ra:  -200 -0.032% RA6M2_EK
            nrf:   +28 +0.015% pca10040
            rp2:  -256 -0.050% PICO
           samd:   -12 -0.009% ADAFRUIT_ITSYBITSY_M4_EXPRESS

Signed-off-by: Damien George <damien@micropython.org>
2022-06-20 22:28:18 +10:00
Damien George e85a096302 py/emit: Remove logic to detect last-emit-was-return-value.
This optimisation to remove dead code is not as good as it could be.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-20 22:28:18 +10:00
Damien George 0db046b67b py/vm: Change comparison for finally handler search from > to >=.
The search in these cases should include all finally handlers that are
after the current ip.  If a handler starts at exactly ip then it is
considered "after" the ip.  This can happen when END_FINALLY is followed
immediately by a finally handler (from a different finally).

Consider the function:

    def f():
        try:
            return 0
        finally:
            print(1)

The current bytecode emitter generates the following code:

    00 SETUP_FINALLY 5
    02 LOAD_CONST_SMALL_INT 0
    03 RETURN_VALUE
    04 LOAD_CONST_NONE              ****
    05 LOAD_GLOBAL print
    07 LOAD_CONST_SMALL_INT 1
    08 CALL_FUNCTION n=1 nkw=0
    10 POP_TOP
    11 END_FINALLY
    12 LOAD_CONST_NONE
    13 RETURN_VALUE

The LOAD_CONST_NONE marked with **** is dead code because it follows a
RETURN_VALUE, and nothing jumps to this LOAD_CONST_NONE.  If the emitter
could remove this this dead code it would produce:

    00 SETUP_FINALLY 4
    02 LOAD_CONST_SMALL_INT 0
    03 RETURN_VALUE
    04 LOAD_GLOBAL print
    06 LOAD_CONST_SMALL_INT 1
    07 CALL_FUNCTION n=1 nkw=0
    09 POP_TOP
    10 END_FINALLY
    11 LOAD_CONST_NONE
    12 RETURN_VALUE

In this case the finally block (which starts at offset 4) immediately
follows the RETURN_VALUE.  When RETURN_VALUE executes ip will point to
offset 4 in the bytecode (because the dispatch of the opcode does *ip++)
and so the finally handler will only be found if a >= comparison is used.

It's a similar story for break/continue:

    while True:
        try:
            break
        finally:
            print(1)

Although technically in this case the > comparison still works because the
extra byte from the UNWIND_JUMP (encoding the number of exception handlers
to unwind) doesn't have a *ip++ (just a *ip) so ip remains pointing within
the UNWIND_JUMP opcode, and not at the start of the following finally
handler.  Nevertheless, the change is made to use >= for consistency with
the RETURN_VALUE change.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-20 22:28:18 +10:00
Damien George 9b486340da all: Bump version to 1.19.1.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-17 12:57:59 +10:00
Damien George d7919ea71e all: Bump version to 1.19.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-16 15:11:02 +10:00
Phil Howard 37d5114cec py/makemoduledefs.py: Emit useful error for legacy MP_REGISTER_MODULE.
Catch calls to legacy:

MP_REGISTER_MODULE(name, module, enable)

Emit a friendly error suggesting they be rewritten to:

MP_REGISTER_MODULE(name, module).

Signed-off-by: Phil Howard <phil@pimoroni.com>
2022-06-14 15:05:37 +01:00
Damien George 0e556f22a2 py/dynruntime: Add macros to access more types and mp_const_empty_bytes.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-10 16:42:43 +10:00
Jeremy Herbert 148d12252b py/dynruntime: Add macros to create a new dict and store to dicts. 2022-06-10 16:42:43 +10:00
Damien George f63b4f85aa py/parse: Work around xtensa esp-2020r3 compiler bug.
This commit works around a bug in xtensa-esp32-elf-gcc version esp-2020r3.

The bug is in generation of loop constructs.  The below code is generated
by the xtensa-esp32 compiler.  The first extract is the buggy machine code
and the second extract is the corrected machine code.  The test
`basics/logic_constfolding.py` fails with the first code and succeeds with
the second.

Disassembly of section .text.push_result_rule:

00000000 <push_result_rule>:
  ...
  d6:   209770       or      a9, a7, a7
  d9:   178976       loop    a9, f4 <push_result_rule+0xf4>
                     d9: R_XTENSA_SLOT0_OP   .text.push_result_rule+0xf4
  dc:   030190       rsr.lend        a9
  df:   130090       wsr.lbeg        a9
  e2:   a8c992       addi    a9, a9, -88
  e5:   06d992       addmi   a9, a9, 0x600
  e8:   130190       wsr.lend        a9
  eb:   002000       isync
  ee:   030290       rsr.lcount      a9
  f1:   01c992       addi    a9, a9, 1
  f4:   1494e7       bne     a4, a14, 10c <push_result_rule+0x10c>
                     f4: R_XTENSA_SLOT0_OP   .text.push_result_rule+0x10c

Disassembly of section .text.push_result_rule:

00000000 <push_result_rule>:
  ...
  d6:   209770       or      a9, a7, a7
  d9:   178976       loop    a9, f4 <push_result_rule+0xf4>
                     d9: R_XTENSA_SLOT0_OP   .text.push_result_rule+0xf4
  dc:   030190       rsr.lend        a9
  df:   130090       wsr.lbeg        a9
  e2:   000091       l32r    a9, fffc00e4 <push_result_rule+0xfffc00e4>
                     e2: R_XTENSA_SLOT0_OP   .literal.push_result_rule+0x18
  e5:   0020f0       nop
  e8:   130190       wsr.lend        a9
  eb:   002000       isync
  ee:   030290       rsr.lcount      a9
  f1:   01c992       addi    a9, a9, 1
  f4:   1494e7       bne     a4, a14, 10c <push_result_rule+0x10c>
                     f4: R_XTENSA_SLOT0_OP   .text.push_result_rule+0x10c

Work done in collaboration with @jimmo.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-09 13:56:30 +10:00
Damien George c7271a86ca py/makemoduledefs.py: Remove shebang line and adjust style of comment.
This file is not executable so shouldn't have the shebang line.  This line
can cause issues when building on Windows msvc when the PyPython variable
is set to something other than "python", because it reverts back to using
the shebang line.

The top comment is also changed to """ style which matches all other
preprocessing scripts in the py/ directory.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-08 15:00:59 +10:00
Damien George cbad559366 py/compile: Give the compiler a hint about num nodes being non-zero.
Without this, newer versions of gcc (eg 11.2.0) used with -O2 can warn
about `q_ptr` being maybe uninitialized, because it doesn't know that there
is at least one qstr being written in to this (alloca'd) memory.

As part of this, change the type of `n` to `size_t` so the compiler knows
it's unsigned and can generate better code.

Code size change for this commit:

       bare-arm:   -28 -0.049%
    minimal x86:    -4 -0.002%
       unix x64:    +0 +0.000%
    unix nanbox:   -16 -0.003%
          stm32:   -24 -0.006% PYBV10
         cc3200:   -32 -0.017%
        esp8266:    +8 +0.001% GENERIC
          esp32:   -52 -0.003% GENERIC
            nrf:   -24 -0.013% pca10040
            rp2:   -32 -0.006% PICO
           samd:   -28 -0.020% ADAFRUIT_ITSYBITSY_M4_EXPRESS

Signed-off-by: Damien George <damien@micropython.org>
2022-06-08 14:59:43 +10:00
Damien George f506bf342a py/bc: Remove unused mp_opcode_format function.
This was made redundant by f2040bfc7e, which
also did not update this function for the change to qstr-opcode encoding,
so it does not work correctly anyway.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:32:38 +10:00
Damien George b37b578214 py/persistentcode: Remove remaining native qstr linking support.
Support for architecture-specific qstr linking was removed in
d4d53e9e11, where native code was changed to
access qstr values via qstr_table.  The only remaining use for the special
qstr link table in persistentcode.c is to support native module written in
C, linked via mpy_ld.py.  But native modules can also use the standard
module-level qstr_table (and obj_table) which was introduced in the .mpy
file reworking in f2040bfc7e.

This commit removes the remaining native qstr liking support in
persistentcode.c's load_raw_code function, and adds two new relocation
options for constants.qstr_table and constants.obj_table.  mpy_ld.py is
updated to use these relocations options instead of the native qstr link
table.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:19:55 +10:00
Andrew Leech 7d9cc69645 rp2/Makefile: Use cmake for "make submodules" task when needed.
Because the submodule list can be updated by cmake files.

Signed-off-by: Andrew Leech <andrew@alelec.net>
2022-06-03 14:29:11 +10:00
Damien George efe23aca71 all: Remove third argument to MP_REGISTER_MODULE.
It's no longer needed because this macro is now processed after
preprocessing the source code via cpp (in the qstr extraction stage), which
means unused MP_REGISTER_MODULE's are filtered out by the preprocessor.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-02 16:31:37 +10:00
Damien George 47f634300c py: Change makemoduledefs process so it uses output of qstr extraction.
This cleans up the parsing of MP_REGISTER_MODULE() and generation of
genhdr/moduledefs.h so that it uses the same process as compressed error
string messages, using the output of qstr extraction.

This makes sure all MP_REGISTER_MODULE()'s that are part of the build are
correctly picked up.  Previously the extraction would miss some (eg if you
had a mod.c file in the board directory for an stm32 board).

Build speed is more or less unchanged.

Thanks to @stinos for the ports/windows/msvc/genhdr.targets changes.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-02 16:29:53 +10:00
Damien George 4a1ae99ac3 extmod/machine_i2c: Add optional support for write-then-read transfers.
This option is useful for ports where it's more efficient to do a full I2C
transfer in one go.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-01 13:20:27 +10:00
Damien George 4290d51320 py/emitinlinethumb: Make float instruction use dynamically selectable.
This allows mpy-cross to dynamically select whether ARMv7-M float
instructions are supported in @micropython.asm_thumb functions.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-26 12:22:07 +10:00
Damien George cca08922d9 py/emitinlinethumb: Make ARMv7-M instruction use dynamically selectable.
This follows on from a5324a1074 and allows
mpy-cross to dynamically select whether ARMv7-M instructions are supported
in @micropython.asm_thumb functions.

The config option MICROPY_EMIT_INLINE_THUMB_ARMV7M is no longer needed, it
is now controlled by MICROPY_EMIT_THUMB_ARMV7M.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-26 11:54:48 +10:00
Damien George 5956466c0e py/builtin: Clean up and simplify import_stat and builtin_open config.
The following changes are made:

- If MICROPY_VFS is enabled then mp_vfs_import_stat and mp_vfs_open are
  automatically used for mp_import_stat and mp_builtin_open respectively.

- If MICROPY_PY_IO is enabled then "open" is automatically included in the
  set of builtins, and points to mp_builtin_open_obj.

This helps to clean up and simplify the most common port configuration.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-25 13:04:45 +10:00
Damien George aa53d2f84a py/asmthumb: Provide implementations of clz/ctz for msvc.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-24 22:27:18 +10:00
Damien George c1b9d2259e py/dynruntime.mk: Add basic support for armv6m architecture.
The examples/natmod features0 and features1 examples now build and run on
ARMv6-M platforms.  More complicated examples are not yet supported because
the compiler emits references to built-in functions like __aeabi_uidiv.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 23:01:25 +10:00
Damien George 17ac68770c py/persistentcode: Select ARMV6M as maximum when __thumb2__ not defined.
If __thumb2__ is defined by the compiler then .mpy files marked as ARMV6M
and above (up to ARMV7EMDP) are supported.  If it's not defined then only
ARMV6M .mpy files are supported.  This makes sure that on CPUs like
Cortex-M0+ (where __thumb2__ is not defined) only .mpy files marked as
ARMV6M can be imported.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 23:01:25 +10:00
Damien George a5324a1074 py/asmthumb: Make ARMv7-M instruction use dynamically selectable.
This commit adjusts the asm_thumb_xxx functions so they can be dynamically
configured to use ARMv7-M instructions or not.  This is available when
MICROPY_DYNAMIC_COMPILER is enabled, and then controlled by the value of
mp_dynamic_compiler.native_arch.

If MICROPY_DYNAMIC_COMPILER is disabled the previous behaviour is retained:
the functions emit ARMv7-M instructions only if MICROPY_EMIT_THUMB_ARMV7M
is enabled.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 23:01:12 +10:00
Damien George d4d53e9e11 py/emitnative: Access qstr values using indirection table qstr_table.
This changes the native emitter to access qstr values using the qstr
indirection table qstr_table, but only when generating native code that
will be saved to a .mpy file.  This makes the resulting native code fully
static, ie it does not require any fix-ups or rewriting when it is
imported.

The performance of native code is more or less unchanged.  Benchmark
results on PYBv1.0 (using --via-mpy and --emit native) are:

N=100 M=100          baseline -> this-commit     diff      diff% (error%)
bm_chaos.py            407.16 ->     411.85 :   +4.69 =  +1.152% (+/-0.01%)
bm_fannkuch.py         100.89 ->     101.20 :   +0.31 =  +0.307% (+/-0.01%)
bm_fft.py             3521.17 ->    3441.72 :  -79.45 =  -2.256% (+/-0.00%)
bm_float.py           6707.29 ->    6644.83 :  -62.46 =  -0.931% (+/-0.00%)
bm_hexiom.py            55.91 ->      55.41 :   -0.50 =  -0.894% (+/-0.00%)
bm_nqueens.py         5343.54 ->    5326.17 :  -17.37 =  -0.325% (+/-0.00%)
bm_pidigits.py         603.89 ->     632.79 :  +28.90 =  +4.786% (+/-0.33%)
core_qstr.py            64.18 ->      64.09 :   -0.09 =  -0.140% (+/-0.01%)
core_yield_from.py     313.61 ->     311.11 :   -2.50 =  -0.797% (+/-0.03%)
misc_aes.py            654.29 ->     659.75 :   +5.46 =  +0.834% (+/-0.02%)
misc_mandel.py        4205.10 ->    4272.08 :  +66.98 =  +1.593% (+/-0.01%)
misc_pystone.py       3077.79 ->    3128.39 :  +50.60 =  +1.644% (+/-0.01%)
misc_raytrace.py       388.45 ->     393.71 :   +5.26 =  +1.354% (+/-0.01%)
viper_call0.py         576.83 ->     566.76 :  -10.07 =  -1.746% (+/-0.05%)
viper_call1a.py        550.39 ->     540.12 :  -10.27 =  -1.866% (+/-0.11%)
viper_call1b.py        438.32 ->     432.09 :   -6.23 =  -1.421% (+/-0.11%)
viper_call1c.py        442.96 ->     436.11 :   -6.85 =  -1.546% (+/-0.08%)
viper_call2a.py        536.31 ->     527.37 :   -8.94 =  -1.667% (+/-0.04%)
viper_call2b.py        378.99 ->     377.50 :   -1.49 =  -0.393% (+/-0.08%)

Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 15:43:06 +10:00
Damien George 94955e8e3d py/asm: Add ASM_LOAD16_REG_REG_OFFSET macro for load-u16 with offset.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 14:21:16 +10:00
Damien George 8af5e2551f py/asmarm: Add asm_arm_ldrh_reg_reg_offset() helper func.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 14:21:16 +10:00
Damien George 94ae023136 py/asmthumb: Add asm_thumb_ldrh_reg_reg_i12_optimised() helper func.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 14:21:14 +10:00
Damien George 689138d484 py/asmthumb: Fix offset variable name in ldr, ldrh and strh functions.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 14:20:10 +10:00
Damien George 7883ae413d py/emitnative: Provide dedicated local for exception unwind handler ptr.
This eliminates the need to save and restore the exception unwind handler
pointer when calling nlr_push.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-19 17:31:56 +10:00
Damien George b608964804 py/emitnative: Simplify generation of code that loads prelude pointer.
It's possible to use REG_PARENT_ARG_1 instead of REG_LOCAL_3.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-19 17:31:56 +10:00
Damien George 56f2d3c2e5 py/asmthumb: Fix PC relative load by sign extending the constant.
PC relative offsets can be negative, in which case the movw result must be
sign extended.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-19 17:31:56 +10:00
Jim Mussared 8b201dc4c3 py: Remove support for MICROPY_PORT_BUILTIN_MODULES.
This functionality is now replaced with MP_REGISTER_MODULE.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-18 20:57:09 +10:00
Jim Mussared d8d3e6ae78 py: Make builtin modules use MP_REGISTER_MODULE.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-18 20:49:12 +10:00
Jim Mussared 4eab44a1ec extmod: Make extmod modules use MP_REGISTER_MODULE.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-18 20:49:12 +10:00
Jim Mussared 469450171f py/makemoduledefs.py: Allow multiple ways to register a module.
For example, ussl can come from axtls or mbedtls. If neither are enabled
then don't try and set an empty definition twice, and only include it
once in MICROPY_REGISTERED_MODULES.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-18 20:49:12 +10:00
Damien George 079f3e5e5b py/parse: Allow all constant objects to be used in "X = const(o)".
Now that constant tuples are supported in the parser, eg (1, True, "str"),
it's a small step to allow anything that is a constant to be used with the
pattern:

    from micropython import const

    X = const(obj)

This commit makes the required changes to allow the following types of
constants:

    from micropython import const

    _INT = const(123)
    _FLOAT = const(1.2)
    _COMPLEX = const(3.4j)
    _STR = const("str")
    _BYTES = const(b"bytes")
    _TUPLE = const((_INT, _STR, _BYTES))
    _TUPLE2 = const((None, False, True, ..., (), _TUPLE))

Prior to this, only integers could be used in const(...).

Signed-off-by: Damien George <damien@micropython.org>
2022-05-18 16:18:35 +10:00
Damien George 8588525868 py/compile: De-duplicate constant objects in module's constant table.
The recent rework of bytecode made all constants global with respect to the
module (previously, each function had its own constant table).  That means
the constant table for a module is shared among all functions/methods/etc
within the module.

This commit add support to the compiler to de-duplicate constants in this
module constant table.  So if a constant is used more than once -- eg 1.0
or (None, None) -- then the same object is reused for all instances.

For example, if there is code like `print(1.0, 1.0)` then the parser will
create two independent constants 1.0 and 1.0.  The compiler will then (with
this commit) notice they are the same and only put one of them in the
constant table.  The bytecode will then reuse that constant twice in the
print expression.  That allows the second 1.0 to be reclaimed by the GC,
also means the constant table has one less entry so saves a word.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-18 15:23:11 +10:00
Damien George 90682f43af py/compile: Allow new qstrs to be allocated at all compiler passes.
Prior to this commit, all qstrs were required to be allocated (by calling
mp_emit_common_use_qstr) in the MP_PASS_SCOPE pass (the first one).  But
this is an unnecessary restriction, which is lifted by this commit.
Lifting the restriction simplifies the compiler because it can allocate
qstrs in later passes.

This also generates better code, because in some cases (eg when a variable
is closed over) the scope of an identifier is not known until a bit later
and then the identifier no longer needs its qstr allocated in the global
table.

Code size is reduced for all ports with this commit.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 23:39:22 +10:00
Damien George 1fb01bd6c5 py/emitnative: Put a pointer to the native prelude in child_table array.
Some architectures (like esp32 xtensa) cannot read byte-wise from
executable memory.  This means the prelude for native functions -- which is
usually located after the machine code for the native function -- must be
placed in separate memory that can be read byte-wise.  Prior to this commit
this was achieved by enabling N_PRELUDE_AS_BYTES_OBJ for the emitter and
MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ for the runtime.  The prelude was
then placed in a bytes object, pointed to by the module's constant table.

This behaviour is changed by this commit so that a pointer to the prelude
is stored either in mp_obj_fun_bc_t.child_table, or in
mp_obj_fun_bc_t.child_table[num_children] if num_children > 0.  The reasons
for doing this are:

1. It decouples the native emitter from runtime requirements, the emitted
   code no longer needs to know if the system it runs on can/can't read
   byte-wise from executable memory.

2. It makes all ports have the same emitter behaviour, there is no longer
   the N_PRELUDE_AS_BYTES_OBJ option.

3. The module's constant table is now used only for actual constants in the
   Python code.  This allows further optimisations to be done with the
   constants (eg constant deduplication).

Code size change for those ports that enable the native emitter:
   unix x64:   +80 +0.015%
      stm32:   +24 +0.004% PYBV10
    esp8266:   +88 +0.013% GENERIC
      esp32:   -20 -0.002% GENERIC[incl -112(data)]
        rp2:   +32 +0.005% PICO

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 16:44:49 +10:00
Damien George 1762990579 py/bc: Provide separate code-state setup funcs for bytecode and native.
mpy-cross will now generate native code based on the size of
mp_code_state_native_t, and the runtime will use this struct to calculate
the offset of the .state field.  This makes native code generation and
execution (which rely on this struct) independent to the settings
MICROPY_STACKLESS and MICROPY_PY_SYS_SETTRACE, both of which change the
size of the mp_code_state_t struct.

Fixes issue #5059.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 14:25:51 +10:00
Damien George 8e1db993cd py/asmx64: Support full range of regs in asm_x64_lea_disp_to_r64.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 14:25:51 +10:00
Damien George c49d5207e9 py/persistentcode: Remove unicode feature flag from .mpy file.
Prior to this commit, even with unicode disabled .py and .mpy files could
contain unicode characters, eg by entering them directly in a string as
utf-8 encoded.

The only thing the compiler disallowed (with unicode disabled) was using
\uxxxx and \Uxxxxxxxx notation to specify a character within a string with
value >= 0x100; that would give a SyntaxError.

With this change mpy-cross will now accept \u and \U notation to insert a
character with value >= 0x100 into a string (because the -mno-unicode
option is now gone, there's no way to forbid this).  The runtime will
happily work with strings with such characters, just like it already works
with strings with characters that were utf-8 encoded directly.

This change simplifies things because there are no longer any feature
flags in .mpy files, and any bytecode .mpy will now run on any target.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 12:51:54 +10:00
Damien George b295b6f1f3 py/persistentcode: Remove obsolete comment about qstr window size.
This was made obsolete in f2040bfc7e

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 12:51:54 +10:00
Damien George 5b700b0af9 all: Reformat remaining C code that doesn't have a space after a comma.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-05 13:30:40 +10:00
Damien George 1216c9fffa py/objmodule: Move stray #include to top of file.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-05 11:02:38 +10:00
Damien George fca5701f74 py/malloc: Introduce m_tracked_calloc, m_tracked_free functions.
Enabled by MICROPY_TRACKED_ALLOC.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-05 10:31:50 +10:00
Jim Mussared 0e7bfc88c6 all: Use mp_obj_malloc everywhere it's applicable.
This replaces occurences of

    foo_t *foo = m_new_obj(foo_t);
    foo->base.type = &foo_type;

with

    foo_t *foo = mp_obj_malloc(foo_t, &foo_type);

Excludes any places where base is a sub-field or when new0/memset is used.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-03 22:28:14 +10:00
Jim Mussared 6a3bc0e1a1 py/objfloat: Explain why mp_obj_malloc isn't used.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-03 22:25:40 +10:00
Jim Mussared 709e8328d9 py/obj: Introduce mp_obj_malloc macro to allocate, and set object type.
This is to replace the following:

    mp_foo_obj_t *self = m_new_obj(mp_foo_obj_t);
    self->base.type = &mp_type_foo;

with:

    mp_foo_obj_t *self = mp_obj_malloc(mp_foo_obj_t, &mp_type_foo);

Calling the function is less code than inlining setting the type
everywhere, adds up to ~100 bytes on PYBV11.

It also helps to avoid an easy mistake of forgetting to set the type.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-03 22:23:46 +10:00
Damien George 590de399f0 py/emitcommon: Don't implicitly close class vars that are assigned to.
When in a class body or at the module level don't implicitly close over
variables that have been assigned to.

Fixes issue #8603.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-03 16:38:43 +10:00
Damien George 402df833fe py/modsys: Introduce sys.implementation._machine constant.
This contains a string useful for identifying the underlying machine.  This
string is kept consistent with the second part of the REPL banner via the
new config option MICROPY_BANNER_MACHINE.

This makes os.uname() more or less redundant, as all the information in
os.uname() is now available in the sys module.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-28 17:23:03 +10:00
Damien George 59c5d41611 py/modsys: Rename sys.implementation.mpy to sys.implementation._mpy.
Per CPython docs, non-standard attributes must begin with an underscore.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-28 17:23:03 +10:00
Damien George 40047823bc py/modsys: Append MicroPython git version and build date to sys.version.
This commit adds the git hash and build date to sys.version.  This is
allowed according to CPython docs, and is what PyPy does.  The docs state:

    A string containing the version number of the Python interpreter plus
    additional information on the build number and compiler used.

Eg on CPython:

    Python 3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> sys.version
    '3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0]'

and PyPy:

    Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)
    [PyPy 5.6.0 with GCC 6.2.0 20161109] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>>> import sys
    >>>> sys.version
    '2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)\n[PyPy 5.6.0 with GCC ...

With this commit on MicroPython we now have:

    MicroPython v1.18-371-g9d08eb024 on 2022-04-28; linux [GCC 11.2.0] v...
    Use Ctrl-D to exit, Ctrl-E for paste mode
    >>> import sys
    >>> sys.version
    '3.4.0; MicroPython v1.18-371-g9d08eb024 on 2022-04-28'

Note that the start of the banner is the same as the end of sys.version.
This helps to keep code size under control because the string can be reused
by the compiler.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-28 15:23:17 +10:00
Damien George a8f23f6366 shared/readline: Make tab insert an indent when it follows whitespace.
Entering tab at the REPL will now make it insert an indent (4 spaces) in
the following cases:
- after any whitespace on a line
- at the start of a line that is not the first line

This changes the existing behaviour where a tab would insert an indent only
if there were no matches in the auto-complete search, and it was the start
of the line.  This means, if there were any symbols in the global
namespace, tab could never be used to indent.

Note that entering tab at the start of the first line will still do
auto-completion, but will now do nothing if there are no symbols in the
global namespace, which is more consistent than before.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-22 17:00:16 +10:00
Jon Bjarni Bjarnason 1ded8a2977 py/objtype: Convert result of user __contains__ method to bool.
Per https://docs.python.org/3/reference/expressions.html#membership-test-operations

    For user-defined classes which define the contains() method, x in y
    returns True if y.contains(x) returns a true value, and False
    otherwise.

Fixes issue #7884.
2022-04-20 15:44:46 +10:00
Damien George 4ca96983ff py/persistentcode: Support loading and saving tuples in .mpy files.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 23:52:14 +10:00
Damien George 35c0cff92b py/parse: Add MICROPY_COMP_CONST_TUPLE option to build const tuples.
This commit adds support to the parser so that tuples which contain only
constant elements (bool, int, str, bytes, etc) are immediately converted to
a tuple object.  This makes it more efficient to use tuples containing
constant data because they no longer need to be created at runtime by the
bytecode (or native code).

Furthermore, with this improvement constant tuples that are part of frozen
code are now able to be stored fully in ROM (this will be implemented in
later commits).

Code size is increased by about 400 bytes on Cortex-M4 platforms.

See related issue #722.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 23:52:12 +10:00
Damien George 24bc1f61f9 py/parse: Print const object value in mp_parse_node_print.
To give more information when printing the parse tree.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 22:45:42 +10:00
Damien George e52f14d057 py/parse: Factor obj extract code to mp_parse_node_extract_const_object.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 22:44:56 +10:00
Damien George 42d0bd2c17 py/persistentcode: Define enum values for obj types instead of letters.
To keep the separate parts of the code that use these values in sync.  And
make it easier to add new object types.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 22:44:04 +10:00
Damien George b8d959d6cf Revert "py/emitnative: Don't store prelude at end of machine code if..."
This reverts commit 7e8222ae06.

The prelude data must exist somewhere in the native code so load_raw_code
and mpy-tool.py can access and parse it.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 14:06:38 +10:00
Damien George 75506e496f py/scheduler: Add support for scheduling static C-based callbacks.
If MICROPY_SCHEDULER_STATIC_NODES is enabled then C code can declare a
static mp_sched_node_t and schedule a callback using
mp_sched_schedule_node().  In contrast to using mp_sched_schedule(), the
node version will have at most one pending callback outstanding, and will
always be able to schedule if there is nothing already scheduled on this
node.  This guarantees that the the callback will be called exactly once
after it is scheduled.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 12:31:53 +10:00
Christian Zietz b0bcb3862b py/emitinlinethumb: Use 16 bit encodings for PUSH LR and POP PC.
The Thumb instruction set has special 16 bit encodings for PUSH involving
LR and POP involving PC, which are commonly used in nested functions.

Using this encoding is particularly important for ARMv6-M, where the more
general 32 bit encoding of PUSH and POP is unavailable.
2022-04-11 15:35:39 +10:00
Damien George aab005c75b extmod/modusocket: Provide config macro for socket.listen backlog deflt.
To make it possible to change this value for any given port or board.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-11 15:28:56 +10:00
Daniel Jour 8baf05af8c py/makeqstrdefs: Cleanup and extend source file classification.
- The classification of source files in makeqstrdefs.py has been moved into
  functions to consolidate the logic for that classification into a single
  place.
- Classification of source files (into C or C++ or "other" files) is based
  on the filename extension.
- For C++ there are many more common filename extensions than just ".cpp";
  see "Options Controlling the Kind of Output" in man gcc for example.  All
  common extensions for C++ source files which need preprocessing have been
  added.
2022-04-01 15:03:21 +11:00
Damien George 40f5c743db py/runtime: Remove unnecessary check for kw_value == MP_OBJ_NULL.
The values are always real objects, only the key can be MP_OBJ_NULL to
indicate a **kwargs entry.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-01 09:20:42 +11:00
Damien George bd556b6996 py: Fix compiling and decoding of *args at large arg positions.
There were two issues with the existing code:

1. "1 << i" is computed as a 32-bit number so would overflow when
   executed on 64-bit machines (when mp_uint_t is 64-bit).  This meant that
   *args beyond 32 positions would not be handled correctly.

2. star_args must fit as a positive small int so that it is encoded
   correctly in the emitted code.  MP_SMALL_INT_BITS is too big because it
   overflows a small int by 1 bit.  MP_SMALL_INT_BITS - 1 does not work
   because it produces a signed small int which is then sign extended when
   extracted (even by mp_obj_get_int_truncated), and this sign extension
   means that any position arg after *args is also treated as a star-arg.
   So the maximum bit position is MP_SMALL_INT_BITS - 2.  This means that
   MP_OBJ_SMALL_INT_VALUE() can be used instead of
   mp_obj_get_int_truncated() to get the value of star_args.

These issues are fixed by this commit, and a test added.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-01 09:20:42 +11:00
Damien George e3de723e2d py/emitbc: Assert that a small int fits its encoding when emitting one.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-31 23:59:10 +11:00
David Lechner 2e3f2045f9 py/runtime: Use size_t/ssize_t instead of uint/int.
This replaces instances of uint with size_t and int with ssize_t in
the mp_call_prepare_args_n_kw_var() function since all of the variables
are used as array offsets.

Also sort headers while we are touching this.

Signed-off-by: David Lechner <david@pybricks.com>
2022-03-31 17:01:25 +11:00
David Lechner 9b74d71aa7 py/runtime: Drop new_alloc < 4 check.
To reach this check, n_kw has to be >= 1 and therefore args2_alloc has
to be >= 2. Therefore new_alloc will always be >= 4. So this check will
never be true and can be removed.

Signed-off-by: David Lechner <david@pybricks.com>
2022-03-31 17:01:03 +11:00
David Lechner 3679a47eb0 py/runtime: Do not overallocate when len is known.
This fixes overallocating an extra mp_obj_t when the length of *args and
**args is known. Previously we were allocating 1 mp_obj_t for each
n_args and n_kw plus the length of each *arg and **arg (if they are
known). Since n_args includes *args and n_kw includes **args, this was
allocating an extra mp_obj_t in addition to the length of these args
when unpacked.

To fix this, we just subtract 1 from the length to account for the 1
already implicitly allocated by n_args and n_kw.

Signed-off-by: David Lechner <david@pybricks.com>
2022-03-31 17:00:50 +11:00
David Lechner 783b1a868f py/runtime: Allow multiple *args in a function call.
This is a partial implementation of PEP 448 to allow unpacking multiple
star args in a function or method call.

This is implemented by changing the emitted bytecodes so that both
positional args and star args are stored as positional args.  A bitmap is
added to indicate if an argument at a given position is a positional
argument or a star arg.

In the generated code, this new bitmap takes the place of the old star arg.
It is stored as a small int, so this means only the first N arguments can
be star args where N is the number of bits in a small int.

The runtime is modified to interpret this new bytecode format while still
trying to perform as few memory reallocations as possible.

Signed-off-by: David Lechner <david@pybricks.com>
2022-03-31 16:59:30 +11:00
David Lechner 1e99d29f36 py/runtime: Allow multiple **args in a function call.
This is a partial implementation of PEP 448 to allow multiple ** unpackings
when calling a function or method.

The compiler is modified to encode the argument as a None: obj key-value
pair (similar to how regular keyword arguments are encoded as str: obj
pairs).  The extra object that was pushed on the stack to hold a single **
unpacking object is no longer used and is removed.

The runtime is modified to decode this new format.

Signed-off-by: David Lechner <david@pybricks.com>
2022-03-31 16:54:00 +11:00
Damien George bb70874111 py/vm: Prevent array bound warning when using -MP_OBJ_ITER_BUF_NSLOTS.
This warning can happen on clang 13.0.1 building mpy-cross:

../py/vm.c:748:25: error: array index -3 refers past the last possible
  element for an array in 64-bit address space containing 64-bit (8-byte)
  elements (max possible 2305843009213693952 elements)
  [-Werror,-Warray-bounds]
                        sp[-MP_OBJ_ITER_BUF_NSLOTS + 1] = MP_OBJ_NULL;
                        ^  ~~~~~~~~~~~~~~~~~~~~~~~~~~~

Using pointer access instead of array access works around this warning.

Fixes issue #8467.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-31 10:59:55 +11:00
Damien George 7e8222ae06 py/emitnative: Don't store prelude at end of machine code if not needed.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-30 16:32:17 +11:00
Damien George bf3585b33c py/asmxtensa: Fix use of l32i/s32i when offset won't fit in encoding.
This commit adds optimised l32i/s32i functions that select the best load/
store encoding based on the size of the offset, and uses the function when
necessary in code generation.

Without this, ASM_LOAD_REG_REG_OFFSET() could overflow the word offset
(using a narrow encoding), for example when loading the prelude from the
constant table when there are many (>16) constants.

Fixes issue #8458.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-30 16:32:17 +11:00
Damien George df9a412206 py/compile: Only show raw code that is bytecode.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-30 16:31:53 +11:00
stijn 594c753c27 py/bc.h: Fix C++20 compilation with "volatile".
C++20 is deprecating several usages of the volatile keyword so remove this
one affected case in the codebase which causes such warning.
2022-03-30 16:29:25 +11:00
Damien George b312a7abf5 py/builtinimport: Alias sys to usys if import weak links aren't enabled.
The sys module should always be available (if it's compiled in), eg to
change sys.path for importing.  So provide an explicit alias from "sys" to
"usys" so that "import sys" can always work.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-28 16:09:58 +11:00
Damien George 6d11c69983 py: Change jump-if-x-or-pop opcodes to have unsigned offset argument.
These jumps are always forwards, and it's more efficient in the VM to
decode an unsigned argument.  These opcodes are already optimised versions
of the sequence "dup-top pop-jump-if-x pop" so it doesn't hurt generality
to optimise them further.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-28 15:43:09 +11:00
Damien George acd2c5c834 py/emitbc: Add check for bytecode jump offset overflow.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-28 15:41:51 +11:00
Damien George 538c3c0a55 py: Change jump opcodes to emit 1-byte jump offset when possible.
This commit introduces changes:

- All jump opcodes are changed to have variable length arguments, of either
  1 or 2 bytes (previously they were fixed at 2 bytes).  In most cases only
  1 byte is needed to encode the short jump offset, saving bytecode size.

- The bytecode emitter now selects 1 byte jump arguments when the jump
  offset is guaranteed to fit in 1 byte.  This is achieved by checking if
  the code size changed during the last pass and, if it did (if it shrank),
  then requesting that the compiler make another pass to get the correct
  offsets of the now-smaller code.  This can continue multiple times until
  the code stabilises.  The code can only ever shrink so this iteration is
  guaranteed to complete.  In most cases no extra passes are needed, the
  original 4 passes are enough to get it right by the 4th pass (because the
  2nd pass computes roughly the correct labels and the 3rd pass computes
  the correct size for the jump argument).

This change to the jump opcode encoding reduces .mpy files and RAM usage
(when bytecode is in RAM) by about 2% on average.

The performance of the VM is not impacted, at least within measurment of
the performance benchmark suite.

Code size is reduced for builds that include a decent amount of frozen
bytecode.  ARM Cortex-M builds without any frozen code increase by about
350 bytes.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-28 15:41:38 +11:00
Damien George 9e3e67b1d8 py/objgenerator: Fix unused variables when native gen extracts prelude.
Some compilers will warn about unused variables like scope_flags.  So use
MP_BC_PRELUDE_SIG_DECODE() which will silence these warnings.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-25 12:35:49 +11:00
David Lechner 768879f999 py/smallint: Introduce MP_SMALL_INT_BITS macro.
This adds a new MP_SMALL_INT_BITS macro that is a compile-time constant
that contains the number of bits available in an MP_SMALL_INT.

We can use this in place of the runtime function mp_small_int_bits().

Signed-off-by: David Lechner <david@pybricks.com>
2022-03-25 12:23:43 +11:00
Damien George 1692cad673 py/showbc: Remove global variables and make DECODE_PTR work correctly.
The bytecode state variables mp_showbc_code_start and mp_showbc_constants
have been removed and made local variables passed into the various
functions.

As part of this, the DECODE_PTR macro is fixed so it extracts the relevant
pointer from the child_table (a regression introduced in
f2040bfc7e).

Signed-off-by: Damien George <damien@micropython.org>
2022-03-16 11:59:46 +11:00
Damien George 962ad8622e py/parse: Handle check for target small-int size in parser.
This means that all constants for EMIT_ARG(load_const_obj, obj) are created
in the parser (rather than some in the compiler).

Signed-off-by: Damien George <damien@micropython.org>
2022-03-16 00:41:10 +11:00
Damien George 3c7cab4e98 py/parse: Put const bytes objects in parse tree as const object.
Instead of as an intermediate qstr, which may unnecessarily intern the data
of the bytes object.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-16 00:41:10 +11:00
Damien George 65851ebb51 py/parse: Simplify handling of const int parse nodes.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-16 00:00:25 +11:00
Damien George ac2293161e py/modsys: Add optional mutable attributes sys.ps1/ps2 and use them.
This allows customising the REPL prompt strings.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 10:58:33 +11:00