This was missed in 692d36d779. Probably
never noticed because everything enables `MICROPY_GC_CONSERVATIVE_CLEAR`,
but found via ASAN thanks to @gwangmu & @chibinz.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This provides a way to build a non-DEBUG host binary that still has symbols
and debug information.
Document this for the unix port, and update a comment in the unix port
Makefile.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Supported from GCC 8 and up, and Compiler Explorer suggests it works as
expected with Clang since 3.6 (2014).
- Fixes situation where building embedded MicroPython with -O0 and
MICROPY_NLR_X64 crashes at runtime (due to nlr_push pushing the
frame pointer register EBP). Closes#12421.
- Allows removing the macOS tweak to undo pushing EBP onto the stack
in the generated function prelude.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
In CPython, `_thread.start_new_thread()` returns an ID that is the same ID
that is returned by `_thread.get_ident()`. The current MicroPython
implementation of `_thread.start_new_thread()` always returns `None`.
This modifies the required functions to return a value. The native thread
id is returned since this can be used for interop with other functions, for
example, `pthread_kill()` on *nix. `_thread.get_ident()` is also modified
to return the native thread id so that the values match and avoids the need
for a separate `native_id` attribute.
Fixes issue #12153.
Signed-off-by: David Lechner <david@pybricks.com>
Because mpy_ld.py doesn't know the target object representation, it emits
instances of `MP_OBJ_NEW_QSTR(MP_QSTR_Foo)` as const string objects, rather
than qstrs. However this doesn't work for map keys (e.g. for a locals dict)
because the map has all_keys_are_qstrs flag is set (and also auto-complete
requires the map keys to be qstrs).
Instead, emit them as regular qstrs, and make a functioning MP_OBJ_NEW_QSTR
function available (via `native_to_obj`, also used for e.g. making
integers).
Remove the code from mpy_ld.py to emit qstrs as constant strings, but leave
behind the scaffold to emit constant objects in case we want to do use this
in the future.
Strictly this should be a .mpy sub-version bump, even though the function
table isn't changing, it does lead to a change in behavior for a new .mpy
running against old MicroPython. `mp_native_to_obj` will incorrectly return
the qstr value directly as an `mp_obj_t`, leading to unexpected results.
But given that it's broken at the moment, it seems unlikely that anyone is
relying on this, so it's not work the other downsides of a sub-version bump
(i.e. breaking pure-Python modules that use @native). The opposite case of
running an old .mpy on new MicroPython is unchanged, and remains broken in
exactly the same way.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This adds support for the x format code in struct.pack and struct.unpack.
The primary use case for this is ignoring bytes while unpacking. When
interfacing with existing systems, it may often happen that you either have
fields in a struct that aren't properly specified or you simply don't care
about them. Being able to easily skip them is useful.
Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
Eliminate `TypeError` when format string contains no named conversions.
This matches CPython behavior.
Signed-off-by: mcskatkat <mc_skatkat@hotmail.com>
The rp2 port was enabling SSL and had finalizers enabled via the "extra
features" level, but missed explicitly enabling `MICROPY_PY_SSL_FINALISER`
(like esp32, stm32, and mimxrt did).
This commit makes `MICROPY_PY_SSL_FINALISER` default to enabled if
finalizers are enabled, and removes the explicit setting of this for
esp32, stm32, mimxrt (because they all use the "extra features" level).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The only reason that const had to be disabled was to make the test output
match CPython when const was involved. Instead, this commit fixes the test
to handle the lines where const is used.
Also:
- remove the special handling for MICROPY_PERSISTENT_CODE_SAVE in
unix/mpconfigport.h, and make this automatic.
- move the check for MICROPY_PERSISTENT_CODE_SAVE to where it's used (like
we do for other similar checks) and add a comment explaining it.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This allows the cc3200 port to be build with the standard autobuild script
rather than the custom build-cc3200-latest.sh (which is now removed).
This also fixes the path inside the zip file (by using the `-j` flag to
zip).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is difficult to implement on cmake-based ports, and having the list
of variants in mpconfigboard.{cmake,mk} duplicates information that's
already in board.json.
This removes the existing query-variants make target from stm32 & rp2
and the definition of BOARD_VARIANTS from the various board files.
Also renames the cmake variable to MICROPY_BOARD_VARIANT to match other
variables such as MICROPY_BOARD. The make variable stays as
BOARD_VARIANT.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When set, the split heap is automatically extended with new areas on
demand, and shrunk if a heap area becomes empty during a GC pass or soft
reset.
To save code size the size allocation for a new heap block (including
metadata) is estimated at 103% of the failed allocation, rather than
working from the more complex algorithm in gc_try_add_heap(). This appears
to work well except in the extreme limit case when almost all RAM is
exhausted (~last few hundred bytes). However in this case some allocation
is likely to fail soon anyhow.
Currently there is no API to manually add a block of a given size to the
heap, although that could easily be added if necessary.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This commit:
- Breaks up some long lines for readability.
- Fixes a potential macro argument expansion issue.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
A previous commit removed the unix-specific select module implementation
and made unix use the common one.
This commit adds an optimisation so that the system poll function is used
when polling objects that have a file descriptor. With this optimisation
enabled, if code registers both file-descriptor-based objects, and non-
file-descriptor-based objects with select.poll() then the following occurs:
- the system poll is called for all file-descriptor-based objects with a
timeout of 1ms
- then the bare-metal polling implementation is used for remaining objects,
which calls into their ioctl method (which can be in C or Python)
In the case where all objects have file descriptors, the system poll is
called with the full timeout requested by the caller. That makes it as
efficient as possible in the case everything has a file descriptor.
Benefits of this approach:
- all ports use the same select module implementation
- the unix port now supports polling of all objects and matches bare metal
implementations
- it's still efficient for existing cases where only files and sockets are
polled (on unix)
- the bare metal implementation does not change
- polling of SSL objects will now work on unix by calling in to the ioctl
method on SSL objects (this is required for asyncio ssl support)
Note that extmod/vfs_posix_file.c has poll disable when the optimisation is
enabled, because the code is not reachable when the optimisation is used.
Signed-off-by: Damien George <damien@micropython.org>
In applications that use little memory and run GC regularly, the cost of
the sweep phase quickly becomes prohibitives as the amount of RAM
increases.
On an ESP32-S3 with 2 MB of external SPIRAM, for example, a trivial GC
cycle takes a minimum of 40ms, virtually all of it in the sweep phase.
Similarly, on the UNIX port with 1 GB of heap, a trivial GC takes 47 ms,
again virtually all of it in the sweep phase.
This commit speeds up the sweep phase in the case most of the heap is empty
by keeping track of the ID of the highest block we allocated in an area
since the last GC.
The performance benchmark run on PYBV10 shows between +0 and +2%
improvement across the existing performance tests. These tests don't
really stress the GC, so they were also run with gc.threshold(30000) and
gc.threshold(10000). For the 30000 case, performance improved by up to
+10% with this commit. For the 10000 case, performance improved by at
least +10% on 6 tests, and up to +25%.
Signed-off-by: Damien George <damien@micropython.org>
Previously this was explicitly enabled on esp32/stm32/renesas/mimxrt/samd,
but didn't get a default feature level because it wasn't in py/mpconfig.h.
With this commit it's now enabled at the "extra features" level, which adds
rp2, unix-standard, windows, esp8266, webassembly, and some nrf boards.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When MICROPY_SCHEDULER_STATIC_NODES is enabled, the logic is unchanged.
When MICROPY_SCHEDULER_STATIC_NODES is disable, sched_state is now always
initialised to MP_SCHED_IDLE when calling mp_init(). For example, the use
of mp_sched_vm_abort(), if it aborts a running scheduled function, can lead
to the scheduler starting off in a locked state when the runtime is
restarted, and then it stays locked. This commit fixes that case by
resetting sched_state.
Signed-off-by: Damien George <damien@micropython.org>
This provides similar functionality to the former zlib.DecompIO and
especially CPython's gzip.GzipFile for both compression and decompression.
This class can be used directly, and also can be used from Python to
implement (via io.BytesIO) zlib.decompress and zlib.compress, as well as
gzip.GzipFile.
Enable/disable this on all ports/boards that zlib was previously configured
for.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
There are enough places that implement __exit__ by forwarding directly to
mp_stream_close that this saves code size.
For the cases where __exit__ is a no-op, additionally make their
MP_STREAM_CLOSE ioctl handled as a no-op.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This will be replaced with a new deflate module providing the same
functionality, with an optional frozen Python wrapper providing a
replacement zlib module.
binascii.crc32 is temporarily disabled.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Follow-up to 24c02c4eb5 for when
MICROPY_ENABLE_EXTERNAL_IMPORT=0. It now needs to try both extensible and
non-extensible modules.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Prior to this fix, async for assumed the iterator expression was a simple
identifier, and used that identifier as a local to store the intermediate
iterator object. This is incorrect behaviour.
This commit fixes the issue by keeping the iterator object on the stack as
an anonymous local variable.
Fixes issue #11511.
Signed-off-by: Damien George <damien@micropython.org>
The asyncio module now has much better CPython compatibility and
deserves to be just called "asyncio".
This will avoid people having to write `from uasyncio import asyncio`.
Renames all files, and updates port manifests to use the new path. Also
renames the built-in _uasyncio to _asyncio.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
PEP-498 allows for conversion specifiers like !r and !s to convert the
expression declared in braces to be passed through repr() and str()
respectively.
This updates the logic that detects the end of the expression to also stop
when it sees "![rs]" that is either at the end of the f-string or before
the ":" indicating the start of the format specifier. The "![rs]" is now
retained in the format string, whereas previously it stayed on the end
of the expression leading to a syntax error.
Previously: `f"{x!y:z}"` --> `"{:z}".format(x!y)`
Now: `f"{x!y:z}"` --> `"{!y:z}".format(x)`
Note that "!a" is not supported by `str.format` as MicroPython has no
`ascii()`, but now this will raise the correct error.
Updated cpydiff and added tests.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
So that the delegation functions don't need to be put somewhere global,
like in mpconfigport.h. That would otherwise make it hard for extension
modules to use delegation.
Signed-off-by: Damien George <damien@micropython.org>
Starting with 2757acf6, the `top` variable in `nlr_jump()` in
`nlraarch64.c` was assigned to register `x19` by the compiler. However,
the assembly code writes over that register with
ldp x19, x20, [%0, #32]
since `%0` is now `x19`. This causes the next line
ldp lr, x9, [%0, #16]
to load the wrong values.
To fix the issue, we move the value of the `top` variable from an unknown
register to a known register at the beginning of the asm code then only use
known/hard-coded registers after that.
Fixes issue #11754.
Signed-off-by: David Lechner <david@pybricks.com>
Following how mkrules.cmake works. This makes it easy for a port to enable
frozen code, by defining FROZEN_MANIFEST in its Makefile.
Signed-off-by: Damien George <damien@micropython.org>
This is a MicroPython-specific module that existed to support the old
version of uasyncio. It's undocumented and not enabled on all ports and
takes up code size unnecessarily.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Previously sys.path could be modified by append/pop or slice assignment.
This allows `sys.path = [...]`, which can be simpler in many cases, but
also improves CPython compatibility.
It also allows sys.path to be set to a tuple which means that you can
clear sys.path (e.g. temporarily) with no allocations.
This also makes sys.path (and sys.argv for consistency) able to be disabled
via mpconfig. The unix port (and upytesthelper) require them, so they
explicitly verify that they're enabled.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Otherwise you can get into the confusing state where e.g. sys.ps1 is
enabled in config (via `MICROPY_PY_SYS_PS1_PS2`) but still doesn't actually
get enabled.
Also verify that the required delegation options are enabled in modsys.c.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When compiling mpy-cross, there is no `sys` module, and so there will
be no entries in the `mp_builtin_module_delegation_table`.
MSVC doesn't like this, so instead pretend as if the feature isn't
enabled at all.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This replaces the previous QSTR_null entry in the globals dict which could
leak out to Python (e.g. via iteration of mod.__dict__) and could lead to
crashes.
It results in smaller code size at the expense of turning a lookup into a
loop, but the list it is looping over likely only contains one or two
elements.
To allow a module to register its custom attr function it can use the new
`MP_REGISTER_MODULE_DELEGATION` macro.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Updates any includes, and references from Makefiles/CMake.
This essentially reverts what was done long ago in commit
136b5cbd76
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Damien George
Jim Mussared
Ihor Nehrutsa
Angus Gratton
David Lechner
Daniël van de Giessen
mcskatkat
Damien Tournoud
Jared Hancock