Commit Graph

2057 Commits

Author SHA1 Message Date
Damien George ac2293161e py/modsys: Add optional mutable attributes sys.ps1/ps2 and use them.
This allows customising the REPL prompt strings.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 10:58:33 +11:00
Damien George cac939ddc3 py/modsys: Add optional sys.tracebacklimit attribute.
With behaviour as per CPython.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 10:43:21 +11:00
Damien George d470c5a5ba tests/extmod/vfs_posix.py: Only test statvfs if it exists.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 00:41:03 +11:00
Damien George 3440201e2e tests/micropython: Switch from set.pop to raise-0 to test exc strings.
To not rely on sets, which are an optional feature.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-07 16:48:35 +11:00
Damien George 9a8ee6a5df tests/run-tests.py: Include test files ending in _set as set tests.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-07 16:48:35 +11:00
Damien George 7cd166ff92 tests/basics: Add test for creating small-ints in nan-box builds.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-07 15:25:11 +11:00
Damien George c4b8dae438 tests/unix: Add coverage test for freezing various objects.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-28 19:02:58 +11:00
Damien George 414b59d39c qemu-arm: Add tests for freezing viper and asm_thumb code.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:29:02 +11:00
Damien George 0a2895b099 tests/perf_bench: Skip bm_chaos test if random.randrange is unavailable.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:29:02 +11:00
Damien George f2040bfc7e py: Rework bytecode and .mpy file format to be mostly static data.
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing.  They are also
smaller on disk.

But the real benefit of .mpy files comes when they are frozen into the
firmware.  This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device.  These C data structures can be executed in-place, ie directly from
ROM.  This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).

The downside of frozen code is that it requires recompiling and reflashing
the entire firmware.  This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).

This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware.  The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place.  If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.

With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).

The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded.  Instead only a small qstr table needs to be built (and put in RAM)
at import time.  This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory.  Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).

In more detail, in the VM what used to be (schematically):

    qst = DECODE_QSTR_VALUE;

is now (schematically):

    idx = DECODE_QSTR_INDEX;
    qst = qstr_table[idx];

That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values.  Only qstr_table needs to be linked
when the .mpy is loaded.

Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.

The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before

The qstr indirection in the bytecode has only a small impact on VM
performance.  For stm32 on PYBv1.0 the performance change of this commit
is:

diff of scores (higher is better)
N=100 M=100             baseline -> this-commit  diff      diff% (error%)
bm_chaos.py               371.07 ->  357.39 :  -13.68 =  -3.687% (+/-0.02%)
bm_fannkuch.py             78.72 ->   77.49 :   -1.23 =  -1.563% (+/-0.01%)
bm_fft.py                2591.73 -> 2539.28 :  -52.45 =  -2.024% (+/-0.00%)
bm_float.py              6034.93 -> 5908.30 : -126.63 =  -2.098% (+/-0.01%)
bm_hexiom.py               48.96 ->   47.93 :   -1.03 =  -2.104% (+/-0.00%)
bm_nqueens.py            4510.63 -> 4459.94 :  -50.69 =  -1.124% (+/-0.00%)
bm_pidigits.py            650.28 ->  644.96 :   -5.32 =  -0.818% (+/-0.23%)
core_import_mpy_multi.py  564.77 ->  581.49 :  +16.72 =  +2.960% (+/-0.01%)
core_import_mpy_single.py  68.67 ->   67.16 :   -1.51 =  -2.199% (+/-0.01%)
core_qstr.py               64.16 ->   64.12 :   -0.04 =  -0.062% (+/-0.00%)
core_yield_from.py        362.58 ->  354.50 :   -8.08 =  -2.228% (+/-0.00%)
misc_aes.py               429.69 ->  405.59 :  -24.10 =  -5.609% (+/-0.01%)
misc_mandel.py           3485.13 -> 3416.51 :  -68.62 =  -1.969% (+/-0.00%)
misc_pystone.py          2496.53 -> 2405.56 :  -90.97 =  -3.644% (+/-0.01%)
misc_raytrace.py          381.47 ->  374.01 :   -7.46 =  -1.956% (+/-0.01%)
viper_call0.py            576.73 ->  572.49 :   -4.24 =  -0.735% (+/-0.04%)
viper_call1a.py           550.37 ->  546.21 :   -4.16 =  -0.756% (+/-0.09%)
viper_call1b.py           438.23 ->  435.68 :   -2.55 =  -0.582% (+/-0.06%)
viper_call1c.py           442.84 ->  440.04 :   -2.80 =  -0.632% (+/-0.08%)
viper_call2a.py           536.31 ->  532.35 :   -3.96 =  -0.738% (+/-0.06%)
viper_call2b.py           382.34 ->  377.07 :   -5.27 =  -1.378% (+/-0.03%)

And for unix on x64:

diff of scores (higher is better)
N=2000 M=2000        baseline -> this-commit     diff      diff% (error%)
bm_chaos.py          13594.20 ->  13073.84 :  -520.36 =  -3.828% (+/-5.44%)
bm_fannkuch.py          60.63 ->     59.58 :    -1.05 =  -1.732% (+/-3.01%)
bm_fft.py           112009.15 -> 111603.32 :  -405.83 =  -0.362% (+/-4.03%)
bm_float.py         246202.55 -> 247923.81 : +1721.26 =  +0.699% (+/-2.79%)
bm_hexiom.py           615.65 ->    617.21 :    +1.56 =  +0.253% (+/-1.64%)
bm_nqueens.py       215807.95 -> 215600.96 :  -206.99 =  -0.096% (+/-3.52%)
bm_pidigits.py        8246.74 ->   8422.82 :  +176.08 =  +2.135% (+/-3.64%)
misc_aes.py          16133.00 ->  16452.74 :  +319.74 =  +1.982% (+/-1.50%)
misc_mandel.py      128146.69 -> 130796.43 : +2649.74 =  +2.068% (+/-3.18%)
misc_pystone.py      83811.49 ->  83124.85 :  -686.64 =  -0.819% (+/-1.03%)
misc_raytrace.py     21688.02 ->  21385.10 :  -302.92 =  -1.397% (+/-3.20%)

The code size change is (firmware with a lot of frozen code benefits the
most):

       bare-arm:  +396 +0.697%
    minimal x86: +1595 +0.979% [incl +32(data)]
       unix x64: +2408 +0.470% [incl +800(data)]
    unix nanbox: +1396 +0.309% [incl -96(data)]
          stm32: -1256 -0.318% PYBV10
         cc3200:  +288 +0.157%
        esp8266:  -260 -0.037% GENERIC
          esp32:  -216 -0.014% GENERIC[incl -1072(data)]
            nrf:  +116 +0.067% pca10040
            rp2:  -664 -0.135% PICO
           samd:  +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS

As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.

In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place.  Performance is not impacted too much.  Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM.  This will
essentially be able to replace frozen code for most applications.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:08:43 +11:00
stijn ff9c708507 tests/run-tests.py: Skip repl tests when running windows underneath.
Some versions of Python (for instance: the mingw-w64 version which can be
installed on MSYS2) do include a pty module and claim to be posix-like
(os.name == 'posix'), yet the select.select call used in run-tests.py hangs
forever.  To be on the safe side just exclude anything which might be
running on windows.
2022-02-18 15:14:47 +11:00
Damien George 2ea21abae0 tests/extmod/vfs_fat_finaliser.py: Make finalisation more robust.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-12 09:45:32 +11:00
Damien George e8bc4a3a5b tests/run-perfbench.py: Use SKIP consistently, and increase print width.
A script will print "SKIP" if it wants to be skipped, so the test runner
must also use uppercase SKIP.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-11 22:19:38 +11:00
Damien George a434705700 tests/perf_bench: Add perf test for yield-from execution.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-11 13:42:00 +11:00
Damien George 75da124cf8 tests/perf_bench: Add perf tests for qstr interning and importing .mpy.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-10 15:25:33 +11:00
Damien George b33fdbe535 tests/run-perfbench.py: Allow a test to SKIP, and to have a .exp file.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-10 15:25:33 +11:00
iabdalkader 30a022548f tests/multi_net/udp_data.py: Make UDP test more reliable.
The current test depends on a specific number and order of packets to pass,
which can't be reproduced every run due to the unreliable UDP protocol.
This patch adds simple packets sequencing, retransmits with timeouts, and a
packet loss threshold, to make the test more tolerant to UDP protocol
packet drops and reordering.
2022-02-09 14:05:01 +11:00
Damien George ab2923dfa1 all: Update Python formatting to latest Black version 22.1.0.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-02 16:49:55 +11:00
Christian Decker 2e3a2785cd extmod/modubinascii: Add newline keyword to b2a_base64 function.
This allows encoding things (eg a Basic-Auth header for a request) without
slicing the \n from the string, which allocates additional memory.

Co-authored-by: David Lechner <david@lechnology.com>
2022-01-23 10:18:01 +11:00
stijn dd6967202a py/modmath: Add math.tau, math.nan and math.inf constants.
Configurable by the new MICROPY_PY_MATH_CONSTANTS option.
2022-01-23 09:28:33 +11:00
iabdalkader 6e8f4eaa52 tests/multi_net/udp_data.py: Allow reusing port before bind. 2022-01-21 13:34:33 +11:00
iabdalkader e6ddda29ca tests/multi_net: Close accepted sockets when tests are done.
gc_sweep_all() cleans up sockets via the finaliser, but tests should
cleanly free resources they use.
2022-01-21 13:34:20 +11:00
Jeff Epler 037b2c72a1 py/objstr: Support '{:08}'.format("Jan") like Python 3.10.
The new test has an .exp file, because it is not compatible with Python 3.9
and lower.

See CPython version of the issue at https://bugs.python.org/issue27772

Signed-off-by: Jeff Epler <jepler@gmail.com>
2022-01-19 15:34:32 +11:00
Damien George 2c9dc5742a tests/multi_net: Add testing key/cert to SSL server/client test.
So that this tests works with mbedtls.

Signed-off-by: Damien George <damien@micropython.org>
2022-01-17 17:35:04 +11:00
Damien George c54717a78f tests/run-multitests.py: Set HOST_IP so tests work between PC and board.
Signed-off-by: Damien George <damien@micropython.org>
2022-01-17 17:35:04 +11:00
Damien George 5df1d8be6c tests/run-multitests.py: Ignore lld_pdu_get_tx_flush_nb msgs from IDF.
BLE still functions correctly even though these messages are sometimes
printed by the IDF.  Ignoring them allows the multi_bluetooth tests to pass
on an esp32 board.

Signed-off-by: Damien George <damien@micropython.org>
2022-01-17 14:23:24 +11:00
stijn 19d949a866 tests/extmod: Skip uselect_poll_udp when poll() is not available.
This is the same fix as applied in uselect_poll_basic.py.
2022-01-04 15:07:45 +11:00
Damien George aac5a97d08 ports: Move '.frozen' to second entry in sys.path.
In commit 86ce442607 the '.frozen' entry was
added at the start of sys.path, to allow control over when frozen modules
are searched during import, and retain existing behaviour whereby frozen
was searched before the filesystem.

But Python semantics of sys.path require sys.path[0] to be the directory of
the currently executing script, or ''.

This commit moves the '.frozen' entry to second place in sys.path, so
sys.path[0] retains its correct value (described above).

Signed-off-by: Damien George <damien@micropython.org>
2021-12-29 23:55:36 +11:00
Damien George c768704cfd tests/basics/int_big_cmp.py: Add more tests for big-int comparison.
To improve coverage of mpz_cmp and mpn_cmp.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-21 18:00:11 +11:00
Damien George 2c139bbf4e py/mpz: Fix bugs with bitwise of -0 by ensuring all 0's are positive.
This commit makes sure that the value zero is always encoded in an mpz_t as
neg=0 and len=0 (previously it was just len=0).

This invariant is needed for some of the bitwise operations that operate on
negative numbers, because they cannot handle -0.  For example
(-((1<<100)-(1<<100)))|1 was being computed as -65535, instead of 1.

Fixes issue #8042.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-21 18:00:05 +11:00
Jim Mussared cc23e99f32 py/modio: Remove io.resource_stream function.
This feature is not enabled on any port, it's not in CPython's io module,
and functionality is better suited to the micropython-lib implementation of
pkg_resources.
2021-12-17 23:53:44 +11:00
Jim Mussared 3770fab334 all: Update Python formatting to latest Black version 21.12b0.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-09 12:09:40 +11:00
Jim Mussared e99f7b6d25 tests/cpydiff: Clarify f-string diffs regarding concatenation.
Concatenation of any literals (including f-strings) should be avoided.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-11-25 22:09:59 +11:00
Damien George 11ed94797d py/lexer: Support nested [] and {} characters within f-string params.
Signed-off-by: Damien George <damien@micropython.org>
2021-11-25 21:50:58 +11:00
Damien George 78ab2eeda3 py/showbc: Print unary-op string when dumping bytecode.
Signed-off-by: Damien George <damien@micropython.org>
2021-11-19 17:05:40 +11:00
jc_.kim 19f09414a6 tests/micropython/const.py: Add comment about required config for test.
Expected result of const.py will be matched only when MICROPY_COMP_CONST is
enabled.  For easy understanding, added description at the first of the
test code.
2021-11-17 14:28:20 +11:00
Damien George 43d08688c3 extmod/uasyncio: Fix gather returning exceptions from a cancelled task.
Fixes issue #5882.
2021-11-17 14:11:31 +11:00
Mike Wadsten c3c2c37fbc tests/basics: Add tests for type-checking subclassed exc instances. 2021-10-21 12:42:48 +11:00
Jim Mussared b326edf68c all: Remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE.
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.

This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.

The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).

It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.

For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:

diff of scores (higher is better)
N=2000 M=2000       bccache -> attrmapcache      diff      diff% (error%)
bm_chaos.py        13742.56 ->   13905.67 :   +163.11 =  +1.187% (+/-3.75%)
bm_fannkuch.py        60.13 ->      61.34 :     +1.21 =  +2.012% (+/-2.11%)
bm_fft.py         113083.20 ->  114793.68 :  +1710.48 =  +1.513% (+/-1.57%)
bm_float.py       256552.80 ->  243908.29 : -12644.51 =  -4.929% (+/-1.90%)
bm_hexiom.py         521.93 ->     625.41 :   +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py     197544.25 ->  217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py      8072.98 ->    8198.75 :   +125.77 =  +1.558% (+/-3.22%)
misc_aes.py        17283.45 ->   16480.52 :   -802.93 =  -4.646% (+/-0.82%)
misc_mandel.py     99083.99 ->  128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py    83860.10 ->   82592.56 :  -1267.54 =  -1.511% (+/-2.27%)
misc_raytrace.py   21490.40 ->   22227.23 :   +736.83 =  +3.429% (+/-1.88%)

This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).

The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):

diff of scores (higher is better)
N=2000 M=2000        native -> nat-attrmapcache  diff      diff% (error%)
bm_chaos.py        14130.62 ->   15464.68 :  +1334.06 =  +9.441% (+/-7.11%)
bm_fannkuch.py        74.96 ->      76.16 :     +1.20 =  +1.601% (+/-1.80%)
bm_fft.py         166682.99 ->  168221.86 :  +1538.87 =  +0.923% (+/-4.20%)
bm_float.py       233415.23 ->  265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py         628.59 ->     734.17 :   +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py     225418.44 ->  232926.45 :  +7508.01 =  +3.331% (+/-3.10%)
bm_pidigits.py      6322.00 ->    6379.52 :    +57.52 =  +0.910% (+/-5.62%)
misc_aes.py        20670.10 ->   27223.18 :  +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py    138221.11 ->  152014.01 : +13792.90 =  +9.979% (+/-2.46%)
misc_pystone.py    85032.14 ->  105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py   19800.01 ->   23350.73 :  +3550.72 = +17.933% (+/-2.79%)

In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.

See #7680 for further discussion.  And see also #7653 for a discussion
about simplifying mpy-cross options.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 16:04:03 +10:00
Damien George 426785a19e py/emitnative: Ensure load_subscr does not clobber existing REG_RET.
Fixes issue #7782, and part of issue #6314.

Signed-off-by: Damien George <damien@micropython.org>
2021-09-13 22:30:24 +10:00
Damien George c0761d28fc tests/perf_bench: Use math.log instead of math.log2.
So MICROPY_PY_MATH_SPECIAL_FUNCTIONS is not needed for these performance
tests.

Signed-off-by: Damien George <damien@micropython.org>
2021-09-13 18:27:39 +10:00
Damien George 08ff71dfcd tests/pybnative: Make while.py test run on boards without pyb.delay.
Signed-off-by: Damien George <damien@micropython.org>
2021-09-01 00:43:41 +10:00
Damien George 8c4ba575fd tests/basics: Split f-string debug printing to separate file with .exp.
This feature {x=} was introduced in Python 3.8 so needs a separate .exp
file to run on earlier Python versions.

See https://bugs.python.org/issue36817

Signed-off-by: Damien George <damien@micropython.org>
2021-08-26 23:56:02 +10:00
Peter Hinch 2296df0a32 extmod/modframebuf: Enable blit between different formats via a palette.
This achieves a substantial performance improvement when rendering glyphs
to color displays, the benefit increasing proportional to the number of
pixels in the glyph.
2021-08-25 15:31:23 +10:00
Jim Mussared 91a99fcf0e tests/extmod/vfs_fat_finaliser.py: Ensure alloc at never-used GC blocks.
Prevents the finaliser from being missed if there's a dangling reference
on the stack to one of the blocks for the files (that this test checks
that they get finalised).

See github.com/micropython/micropython/pull/7659#issuecomment-899479793

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-17 11:20:58 +10:00
Jim Mussared c70930fb24 tests/multi_bluetooth/ble_subscribe.py: Add test for subscription.
This tests both sending indications/notifications from a server to
subscribed clients via gatts_write(...,send_update=True) and subscribing
from a client.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-14 22:44:47 +10:00
Jim Mussared 692d36d779 py: Implement partial PEP-498 (f-string) support.
This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.

It is implemented in the lexer as a syntax translation to `str.format`:
  f"{a}" --> "{}".format(a)

It also supports:
  f"{a=}" --> "a={}".format(a)

This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.

There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
  `SyntaxError: raw f-strings are not supported`.

- literal concatenation of f-strings with adjacent strings will fail
    "{}" f"{a}" --> "{}{}".format(a)    (str.format will incorrectly use
                                         the braces from the non-f-string)
    f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)

- PEP-498 requires the full parser to understand the interpolated
  argument, however because this entirely runs in the lexer it cannot
  resolve nested braces in expressions like
    f"{'}'}"

- The !r, !s, and !a conversions are not supported.

Includes tests and cpydiffs.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-14 16:58:40 +10:00
Damien George 8fcdb5490c extmod/modlwip: Fix close and clean up of UDP and raw sockets.
The correct callback-deregister functions must be called dependent on the
socket type, otherwise resources may not be freed correctly.

Signed-off-by: Damien George <damien@micropython.org>
2021-08-13 23:46:11 +10:00
Damien George 90d47ee34d tests/run-multitests.py: Add broadcast and wait facility.
Test instances can now use the following methods to synchronise their
execution:

    multitest.broadcast("sync message")
    multitest.wait("sync message")

Signed-off-by: Damien George <damien@micropython.org>
2021-08-13 23:26:34 +10:00
Peter Züger d290f369d0 tests/extmod/ujson: Add tests for dump/dumps separators argument.
Basically just copied ujson_dump(s).py and added various valid/invalid
separator tests.

Signed-off-by: Peter Züger <zueger.peter@icloud.com>
2021-08-07 13:52:16 +10:00
Jim Mussared 4e39ff221a py/runtime: Fix bool unary op for subclasses of native types.
Previously a subclass of a type that didn't implement unary_op, or didn't
handle MP_UNARY_OP_BOOL, would raise TypeError on bool conversion.

Fixes #5677.
2021-07-23 12:40:00 +10:00
Jim Mussared 0e3752e82a py/emitnative: Ensure stack settling is safe mid-branch.
And add a test for the case where REG_RET could be in use.

Fixes #7523.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-07-19 23:18:59 +10:00
David P f365025c9c stm32: Replace master/slave with controller/peripheral in I2C and SPI.
Replace "master" with "controller" and "slave" with "peripheral" in
comments, errors, and debug messages.

Add CONTROLLER and PERIPHERAL constants to pyb.SPI and pyb.I2C classes;
retain MASTER and SLAVE constants for backward compatiblity.
2021-07-18 11:23:41 +10:00
Damien George bb00125aaa py: Support single argument to optimised MP_OBJ_STOP_ITERATION.
The MP_OBJ_STOP_ITERATION optimisation is a shortcut for creating a
StopIteration() exception object, and means that heap memory does not need
to be allocated for the exception (in cases where it can be used).  This
commit allows this optimised object to take an optional argument (before,
it could only have no argument).

The commit also adds some new tests to cover corner cases with
StopIteration and generators that previously did not work.

Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
Damien George e3825e28e6 py/objexcept: Make mp_obj_exception_get_value support subclassed excs.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
David Lechner cd506d6220 tests/cpydiff/modules_struct_whitespace_in_format: Run black.
This test snuck through without proper formatting and is causing CI for
other unrelated changes to fail.

Signed-off-by: David Lechner <david@pybricks.com>
2021-07-06 18:19:55 -05:00
Tom McDermott c1f74b3005 docs/library: Warn that ustruct doesn't handle spaces in format strings.
And also add a test to capture the CPython difference.
2021-07-06 14:59:50 +10:00
David Lechner 58e4d72338 py/objexcept: Pretty print OSError also when it has 2 arguments.
This extends pretty-printing of OSError's to handle two arguments when the
exception name is known.

Signed-off-by: David Lechner <david@pybricks.com>
2021-07-01 13:23:54 +10:00
Damien George 7ec95c2768 extmod/uasyncio: Get addr and bind server socket before creating task.
Currently when using uasyncio.start_server() the socket configuration is
done inside a uasyncio.create_task() background function.  If the address
and port are already in use however this throws an OSError which cannot be
cleanly caught behind the create_task().

This commit moves the getaddrinfo and socket binding to the start_server()
function, and only creates the task if that succeeds.  This means that any
OSError from the initial socket configuration is propagated directly up the
call stack, compatible with CPython behaviour.

See #7444.

Signed-off-by: Damien George <damien@micropython.org>
2021-06-26 22:30:22 +10:00
Damien George 180c54d6cc tests/extmod: Make uasyncio_heaplock test more deterministic.
This helps the test pass on systems with an inaccurate sleep time.

Signed-off-by: Damien George <damien@micropython.org>
2021-06-25 11:31:00 +10:00
Damien George adf35cbab0 tests/float: Make bytes/bytearray construct tests work with obj repr C.
2.5 can be represented correctly in object representation C, but 2.3 cannot
(it is slightly truncated).

Signed-off-by: Damien George <damien@micropython.org>
2021-06-18 14:16:07 +10:00
Damien George 514bf1a191 extmod/uasyncio: Fix race with cancelled task waiting on finished task.
This commit fixes a problem with a race between cancellation of task A and
completion of task B, when A waits on B.  If task B completes just before
task A is cancelled then the cancellation of A does not work.  Instead,
the CancelledError meant to cancel A gets passed through to B (that's
expected behaviour) but B handles it as a "Task exception wasn't retrieved"
scenario, printing out such a message (this is because finished tasks point
their "coro" attribute to themselves to indicate they are done, and
implement the throw() method, but that method inadvertently catches the
CancelledError).  The correct behaviour is for B to bounce that
CancelledError back out.

This bug is mainly seen when wait_for() is used, and in that context the
symptoms are:
- occurs when using wait_for(T, S), if the task T being waited on finishes
  at exactly the same time as the wait-for timeout S expires
- task T will have run to completion
- the "Task exception wasn't retrieved message" is printed with
  "<class 'CancelledError'>" as the error (ie no traceback)
- the wait_for(T, S) call never returns (it's never put back on the
  uasyncio run queue) and all tasks waiting on this are blocked forever
  from running
- uasyncio otherwise continues to function and other tasks continue to be
  scheduled as normal

The fix here reworks the "waiting" attribute of Task to be called "state"
and uses it to indicate whether a task is: running and not awaited on,
running and awaited on, finished and not awaited on, or finished and
awaited on.  This means the task does not need to point "coro" to itself to
indicate finished, and also allows removal of the throw() method.

A benefit of this is that "Task exception wasn't retrieved" messages can go
back to being able to print the name of the coroutine function.

Fixes issue #7386.

Signed-off-by: Damien George <damien@micropython.org>
2021-06-16 13:02:37 +10:00
Mike Teachman b0b8ebc4f6 extmod/uasyncio: Add readinto() method to Stream class.
With docs and a multi-test using TCP server/client.

This method is a MicroPython extension, although there is discussion of
adding it to CPython: https://bugs.python.org/issue41305

Signed-off-by: Mike Teachman <mike.teachman@gmail.com>
2021-06-15 13:13:35 +10:00
Zoltán Vörös c4ed17ff34 tests/cpydiff: Add test for array constructor with overflowing value. 2021-06-13 10:30:14 +10:00
Damien George 20a8f4f7ec tests/unix: Add ffi test for integer types.
Signed-off-by: Damien George <damien@micropython.org>
2021-06-06 22:52:25 +10:00
Damien George 7842085434 tests/multi_bluetooth/ble_gap_advertise.py: Allow to work without set.
Signed-off-by: Damien George <damien@micropython.org>
2021-06-06 21:58:07 +10:00
Damien George da8e47da21 tests/run-multitests.py: Allow to work without sys.stdout on target.
Signed-off-by: Damien George <damien@micropython.org>
2021-06-06 21:58:07 +10:00
Damien George 53519e322a py/builtinimport: Change relative import's ValueError to ImportError.
Following CPython change, see https://bugs.python.org/issue37444.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-30 19:35:03 +10:00
Damien George c3199f5649 extmod/modurandom: Support an argument of bits=0 to getrandbits.
This was changed in CPython 3.9; see https://bugs.python.org/issue40282.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-30 17:05:56 +10:00
Macarthur Inbody 34d4dab683 extmod/modurandom: Add error message when getrandbits has bad value.
The random module's getrandbits() method didn't give a proper error message
when calling it with a value that was outside of the range of 1-32, which
can lead to confusion using this function (which under CPython can accept
numbers larger than 32).  Now instead of simply giving a ValueError it
gives an error message that states that the number of bits is constrained.

Also, since the random module's functions getrandbits() and randint()
differ from CPython, tests have been added to describe these differences.
For getrandbits the relevant documentation is shown and added to the docs.
The same is given for randint method so that the information is more easily
found.

Finally, since the int object lacks the bit_length() method there is a test
for that method also to include within the docs, showing the difference to
CPython.
2021-05-30 16:41:30 +10:00
Damien George 025e4b6fbc tests/basics: Split out literal tests that raise SyntaxWarning on CPy.
Fixes issue #7330.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-30 13:41:37 +10:00
Jeff Epler 486fe71c6e tests/extmod/btree_gc.py: Close the database to avoid a memory leak.
Signed-off-by: Jeff Epler <jepler@gmail.com>
2021-05-30 11:50:51 +10:00
Jeff Epler f2dbc91022 py/compile: Raise an error on async with/for outside an async function.
A simple reproducer is:

   async for x in (): x

Before this change, it would cause an assertion error in mpy-cross and
micropython-coverage.
2021-05-30 10:38:48 +10:00
Damien George 6a127810c0 extmod/moduhashlib: Put hash obj in final state after digest is called.
If digest is called then the hash object is put in a "final" state and
calling update() or digest() again will raise a ValueError (instead of
silently producing the wrong result).

See issue #4119.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-26 21:44:46 +10:00
Damien George dc86e04476 tests: Make float and framebuf tests skip or run on big-endian archs.
Signed-off-by: Damien George <damien@micropython.org>
2021-05-26 16:33:18 +10:00
Damien George 5176a2d732 py/emitnative: Fix x86-64 emitter to generate correct 8/16-bit stores.
Fixes issue #6643.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-20 23:43:25 +10:00
Damien George 6d2680fa36 py/objarray: Fix constructing a memoryview from a memoryview.
Fixes issue #7261.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-18 10:18:56 +10:00
Damien George 47e6c52f0c tests/cpydiff: Add test and workaround for function.__module__ attr.
MicroPython does not store any reference from a function object to the
module it was defined in, but there is a way to use function.__globals__ to
indirectly get the module.

See issue #7259.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-16 11:06:46 +10:00
Jeff Epler 94a3f8a4b0 tests/run-tests.py: Parallelize running tests by default.
This significantly reduces the time taken to run the test suite (on the
unix port).  Use `-j1` to disable this feature.

Signed-off-by: Jeff Epler <jepler@gmail.com>
2021-05-14 11:33:31 +10:00
stijn 09be0c083c py/objarray: Implement more/less comparisons for array. 2021-05-13 22:16:14 +10:00
stijn 57365d8557 py/objarray: Prohibit comparison of mismatching types.
Array equality is defined as each element being equal but to keep
code size down MicroPython implements a binary comparison.  This
can only be used correctly for elements with the same binary layout
though so turn it into an NotImplementedError when comparing types
for which the binary comparison yielded incorrect results: types
with different sizes, and floating point numbers because nan != nan.
2021-05-13 22:16:14 +10:00
Damien George 6affcb0104 tests/run-multitests.py: Flush stdout for each line of trace output.
Signed-off-by: Damien George <damien@micropython.org>
2021-05-13 16:26:07 +10:00
Damien George 18d984c8b2 tests/run-perfbench.py: Fix native feature check.
This was broken by 8459f538eb

Signed-off-by: Damien George <damien@micropython.org>
2021-05-11 23:45:36 +10:00
Damien George 4cdcbdb753 tests/thread: Make exc1,exit1,exit2,stacksize1,start1 tests run on rp2.
The RP2040 has 2 cores and supports running at most 2 Python threads (the
main one plus another), and will raise OSError if a thread cannot be
created because core1 is already in use.  This commit adjusts some thread
tests to be robust against such OSError's.  These tests now pass on rp2
boards.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-10 13:07:16 +10:00
Damien George b6b39bff47 py/gc: Make gc_lock_depth have a count per thread.
This commit makes gc_lock_depth have one counter per thread, instead of one
global counter.  This makes threads properly independent with respect to
the GC, in particular threads can now independently lock the GC for
themselves without locking it for other threads.  It also means a given
thread can run a hard IRQ without temporarily locking the GC for all other
threads and potentially making them have MemoryError exceptions at random
locations (this really only occurs on MCUs with multiple cores and no GIL,
eg on the rp2 port).

The commit also removes protection of the GC lock/unlock functions, which
is no longer needed when the counter is per thread (and this also fixes the
cas where a hard IRQ calling gc_lock() may stall waiting for the mutex).

It also puts the check for `gc_lock_depth > 0` outside the GC mutex in
gc_alloc, gc_realloc and gc_free, to potentially prevent a hard IRQ from
waiting on a mutex if it does attempt to allocate heap memory (and putting
the check outside the GC mutex is now safe now that there is a
gc_lock_depth per thread).

Signed-off-by: Damien George <damien@micropython.org>
2021-05-10 13:07:16 +10:00
Damien George 7b923d6c72 tests/thread: Make stress_aes.py test run on bare-metal ports.
This is a long-running test, so make it run in reasonable time on slower,
bare-metal ports.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-08 22:47:03 +10:00
Damien George 9340cfe774 tests/thread: Make stress_create.py test run on esp32.
The esp32 port needs to be idle for finished threads and their resources to
be freed up.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-08 22:47:03 +10:00
Damien George 864e4ecc47 esp32/mpthreadport: Use binary semaphore instead of mutex.
So a lock can be acquired on one Python thread and then released on
another.  A test for this is added.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-08 22:47:03 +10:00
Damien George 47583d8cbd extmod/moductypes: Fix size and offset calculation for ARRAY of FLOAT32.
uctypes.FLOAT32 has a special value representation and
uctypes_struct_scalar_size() should be used instead of GET_SCALAR_SIZE().

Signed-off-by: Damien George <damien@micropython.org>
2021-05-06 13:11:33 +10:00
Damien George 9e29217c73 unix/modffi: Use a union for passing/returning FFI values.
This fixes a bug where double arguments on a 32-bit architecture would not
be passed correctly because they only had 4 bytes of storage (not 8).  It
also fixes a compiler warning/error in return_ffi_value on certian
architectures: array subscript 'double[0]' is partly outside array bounds
of 'ffi_arg[1]' {aka 'long unsigned int[1]'}.

Fixes issue #7064.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-06 12:17:10 +10:00
Artyom Skrobov ca35c0059c py/repl: Autocomplete builtin modules.
Doing "import <tab>" will now complete/list built-in modules.

Originally at adafruit#4548 and adafruit#4608

Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
2021-05-02 23:11:14 +10:00
Kathryn Lingel 1f1a54d0b1 py/repl: Filter private methods from tab completion.
Anything beginning with "_" will now only be tab-completed if there is
already a partial match for such an entry.  In other words, entering
foo.<tab> will no longer complete/list anything beginning with "_".

Originally at adafruit#1850

Signed-off-by: Kathryn Lingel <kathryn@lingel.net>
2021-05-02 23:11:03 +10:00
Damien George 1d9528210b tests/multi_bluetooth: Add performance test for gatt char writes.
Signed-off-by: Damien George <damien@micropython.org>
2021-04-30 16:14:48 +10:00
Damien George 76dab3bf31 tests/run-multitests.py: Provide some convenient serial device shorcuts.
It's now possible to specify a device serial port using shorcuts like:

    $ ./run-multitests.py -i pyb:a0 -i pyb:u1 multi_bluetooth/*.py

Signed-off-by: Damien George <damien@micropython.org>
2021-04-30 15:47:22 +10:00
Damien George 3123f6918b tests: Use .errno instead of .args[0] for OSError exceptions.
Signed-off-by: Damien George <damien@micropython.org>
2021-04-23 22:03:46 +10:00
Damien George 3c4bfd1dec py/objexcept: Support errno attribute on OSError exceptions.
This commit adds the errno attribute to exceptions, so code can retrieve
errno codes from an OSError using exc.errno.

The implementation here simply lets `errno` (and the existing `value`)
attributes work on any exception instance (they both alias args[0]).  This
is for efficiency and to keep code size down.  The pros and cons of this
are:

Pros:
- more compatible with CPython, less difference to document and learn
- OSError().errno will correctly return None, whereas the current way of
  doing it via OSError().args[0] will raise an IndexError
- it reduces code size on most bare-metal ports (because they already have
  the errno qstr)
- for Python code that uses exc.errno the generated bytecode is 2 bytes
  smaller and more efficient to execute (compared with exc.args[0]); so
  bytecode loaded to RAM saves 2 bytes RAM for each use of this attribute,
  and bytecode that is frozen saves 2 bytes flash/ROM for each use
- it's easier/shorter to type, and saves 2 bytes of space in .py files that
  use it (for each use)

Cons:
- increases code size by 4-8 bytes on minimal ports that don't already have
  the `errno` qstr
- all exceptions now have .errno and .value attributes (a cpydiff test is
  added to address this)

See also #2407.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-23 22:03:46 +10:00
Damien George 7d911d2069 tests/net_inet: Add 'Strict-Transport-Security' to exp file.
Because micropython.org now adds this to the headers.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-18 23:20:26 +10:00
Damien George 8459f538eb tests/feature_check: Check for lack of pass result rather than failure.
Commit cb68a5741a broke automatic Python
feature detection when running tests, because some detection relied on a
crash of a feature script returning exactly b"CRASH".

This commit fixes this and improves the situation by testing for the lack
of a known pass result, rather than an exact failure result.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-15 00:52:56 +10:00
stijn a66286f3a0 unix: Improve command line argument processing.
Per CPython everything which comes after the command, module or file
argument is not an option for the interpreter itself.  Hence the processing
of options should stop when encountering those, and the remainder be passed
as sys.argv.  Note the latter was already the case for a module or file but
not for a command.

This fixes issues like 'micropython myfile.py -h' showing the help and
exiting instead of passing '-h' as sys.argv[1], likewise for
'-X <something>' being treated as a special option no matter where it
occurs on the command line.
2021-04-07 12:41:25 +10:00
Jeff Epler 172fb5230a extmod/re1.5: Check and report byte overflow errors in _compilecode.
The generated regex code is limited in the range of jumps and counts, and
this commit checks all cases which can overflow given the right kind of
input regex, and returns an error in such a case.

This change assumes that the results that overflow an int8_t do not
overflow a platform int.

Closes: #7078

Signed-off-by: Jeff Epler <jepler@gmail.com>
2021-04-06 13:36:42 +10:00