Commit Graph

489 Commits

Author SHA1 Message Date
Damien George 80bc9b3dee tools/mpremote: Don't be verbose when using cat command.
Fixes issue #8828.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-08 22:28:26 +10:00
Damien George 7e5137e0ae tools/pyboard.py: Add verbose option to filesystem_command.
Signed-off-by: Damien George <damien@micropython.org>
2022-07-08 22:26:41 +10:00
David Lechner 17f0297c1a tools/ci.sh: Drop ppa requirement for code formatting CI.
The CI scripts were using a PPA to get a backported version of uncrustify
on Ubuntu 20.04.  However, this causes CI to intermittently fail due to
connection issues to launchpad.net or the key server.

Ubuntu 22.04 has a newer version of uncrustify removing the need for the
PPA.  Ubuntu 22.04 is now in beta on GitHub actions, so it can be used.

Signed-off-by: David Lechner <david@pybricks.com>
2022-06-17 16:56:41 +10:00
Maureen Helm bada8b3cad zephyr: Upgrade to Zephyr v3.1.0.
Updates the Zephyr port build instructions and CI to use the latest Zephyr
release tag.

Tested on frdm_k64f.

Signed-off-by: Maureen Helm <maureen.helm@intel.com>
2022-06-17 16:51:34 +10:00
Damien George dea1ba3309 tools/metrics.py: Add mimxrt and renesas-ra to list of available ports.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-17 14:57:58 +10:00
Damien George 1f1afae622 examples/natmod/features3: Add example to test more natmod features.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-10 16:42:43 +10:00
Damien George 17a0d65ee4 tools/mpy_ld.py: Support GOT entries that reference inside mp_fun_table.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-10 16:42:43 +10:00
Damien George 313f082896 tools/ci.sh: Build native .mpy examples for armv7emsp, xtensawin in CI.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-09 13:24:08 +10:00
Damien George abb3850398 tools/mpy_ld.py: Support R_XTENSA_PDIFF32 relocation.
Newer versions of the ESP-IDF's toolchain use this relocation.

Fixes issue #8436.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-09 11:57:22 +10:00
Damien George 6a1dbaad1c tools/ci.sh: Build NUCLEO_H743ZI with -O2 optimisation level.
This tests the build when -O2 is used, which can lead to additional
compiler analysis and warnings.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-08 15:00:59 +10:00
Damien George 5290bfaefe tools/mpremote: Bump version to 0.3.0.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 23:23:14 +10:00
Damien George 646fcdadbf tools/mpremote: Add command to print the version.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 23:22:04 +10:00
Damien George 2fb413b265 tools/mpy-tool.py: Improve generated frozen identifiers.
Frozen identifiers now include their full name hierarchy, eg their class
name.  This makes it easier to understand the generated code.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 16:55:18 +10:00
Damien George 9670a156da all: Rename MICROPY_PY_WIZNET5K to MICROPY_PY_NETWORK_WIZNET5K.
To match MICROPY_PY_NETWORK_CYW43 and MICROPY_PY_NETWORK_NINAW10.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 16:55:18 +10:00
Damien George bf92b0cbf2 tools/ci.sh: Add test for mpy-tool's merging feature.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:51:45 +10:00
Damien George 599a22e569 tools/mpy-tool.py: Rework .mpy merging feature.
Now that the native qstr link table is gone, merging a native .mpy file
with a bytecode .mpy file is not as simple as concatenating the .mpy data.
The qstr_table and obj_table tables from all merged .mpy files must now be
joined together, because they are global to the .mpy file (and hence global
to the merged .mpy file).  This means the bytecode needs to be be decoded,
qstr_table and obj_table indices updated to point to the correct entries in
the new tables, and then the bytecode re-encoded.

This commit makes this change to the merging feature in mpy-tool.py.  This
can now merge an arbitrary number of bytecode .mpy files, and up to one
native .mpy file.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:51:45 +10:00
Damien George f506bf342a py/bc: Remove unused mp_opcode_format function.
This was made redundant by f2040bfc7e, which
also did not update this function for the change to qstr-opcode encoding,
so it does not work correctly anyway.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:32:38 +10:00
Damien George b37b578214 py/persistentcode: Remove remaining native qstr linking support.
Support for architecture-specific qstr linking was removed in
d4d53e9e11, where native code was changed to
access qstr values via qstr_table.  The only remaining use for the special
qstr link table in persistentcode.c is to support native module written in
C, linked via mpy_ld.py.  But native modules can also use the standard
module-level qstr_table (and obj_table) which was introduced in the .mpy
file reworking in f2040bfc7e.

This commit removes the remaining native qstr liking support in
persistentcode.c's load_raw_code function, and adds two new relocation
options for constants.qstr_table and constants.obj_table.  mpy_ld.py is
updated to use these relocations options instead of the native qstr link
table.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:19:55 +10:00
Andrew Leech b92e51efe7 tools/ci.sh: Add build of W5100S_EVB_PICO board to rp2 CI.
This exercises the inclusion of a number of libraries in the rp2 port
including mbedtls and lwip.

Signed-off-by: Andrew Leech <andrew@alelec.net>
2022-06-03 14:35:52 +10:00
Damien George 1d047617bb tools/mpy-tool.py: Remove obsolete unicode flag in .mpy header.
This was removed in c49d5207e9

Signed-off-by: Damien George <damien@micropython.org>
2022-05-26 11:43:46 +10:00
David Lechner d42d35f56d tests/run-tests.py: Enable `-X realtime` option for macOS tests.
This enables the new `-X realtime` runtime option when running tests on
macOS.  This causes MicroPython to configure all threads to be high
priority so that they are allowed to use high precision timers.  This
makes tests that depend on the passage of time more likely to succeed.

CI tests that were disabled because of this are now enabled again.

Signed-off-by: David Lechner <david@pybricks.com>
2022-05-24 00:52:44 +10:00
Damien George c1b9d2259e py/dynruntime.mk: Add basic support for armv6m architecture.
The examples/natmod features0 and features1 examples now build and run on
ARMv6-M platforms.  More complicated examples are not yet supported because
the compiler emits references to built-in functions like __aeabi_uidiv.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 23:01:25 +10:00
Damien George d4d53e9e11 py/emitnative: Access qstr values using indirection table qstr_table.
This changes the native emitter to access qstr values using the qstr
indirection table qstr_table, but only when generating native code that
will be saved to a .mpy file.  This makes the resulting native code fully
static, ie it does not require any fix-ups or rewriting when it is
imported.

The performance of native code is more or less unchanged.  Benchmark
results on PYBv1.0 (using --via-mpy and --emit native) are:

N=100 M=100          baseline -> this-commit     diff      diff% (error%)
bm_chaos.py            407.16 ->     411.85 :   +4.69 =  +1.152% (+/-0.01%)
bm_fannkuch.py         100.89 ->     101.20 :   +0.31 =  +0.307% (+/-0.01%)
bm_fft.py             3521.17 ->    3441.72 :  -79.45 =  -2.256% (+/-0.00%)
bm_float.py           6707.29 ->    6644.83 :  -62.46 =  -0.931% (+/-0.00%)
bm_hexiom.py            55.91 ->      55.41 :   -0.50 =  -0.894% (+/-0.00%)
bm_nqueens.py         5343.54 ->    5326.17 :  -17.37 =  -0.325% (+/-0.00%)
bm_pidigits.py         603.89 ->     632.79 :  +28.90 =  +4.786% (+/-0.33%)
core_qstr.py            64.18 ->      64.09 :   -0.09 =  -0.140% (+/-0.01%)
core_yield_from.py     313.61 ->     311.11 :   -2.50 =  -0.797% (+/-0.03%)
misc_aes.py            654.29 ->     659.75 :   +5.46 =  +0.834% (+/-0.02%)
misc_mandel.py        4205.10 ->    4272.08 :  +66.98 =  +1.593% (+/-0.01%)
misc_pystone.py       3077.79 ->    3128.39 :  +50.60 =  +1.644% (+/-0.01%)
misc_raytrace.py       388.45 ->     393.71 :   +5.26 =  +1.354% (+/-0.01%)
viper_call0.py         576.83 ->     566.76 :  -10.07 =  -1.746% (+/-0.05%)
viper_call1a.py        550.39 ->     540.12 :  -10.27 =  -1.866% (+/-0.11%)
viper_call1b.py        438.32 ->     432.09 :   -6.23 =  -1.421% (+/-0.11%)
viper_call1c.py        442.96 ->     436.11 :   -6.85 =  -1.546% (+/-0.08%)
viper_call2a.py        536.31 ->     527.37 :   -8.94 =  -1.667% (+/-0.04%)
viper_call2b.py        378.99 ->     377.50 :   -1.49 =  -0.393% (+/-0.08%)

Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 15:43:06 +10:00
Damien George 1fb01bd6c5 py/emitnative: Put a pointer to the native prelude in child_table array.
Some architectures (like esp32 xtensa) cannot read byte-wise from
executable memory.  This means the prelude for native functions -- which is
usually located after the machine code for the native function -- must be
placed in separate memory that can be read byte-wise.  Prior to this commit
this was achieved by enabling N_PRELUDE_AS_BYTES_OBJ for the emitter and
MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ for the runtime.  The prelude was
then placed in a bytes object, pointed to by the module's constant table.

This behaviour is changed by this commit so that a pointer to the prelude
is stored either in mp_obj_fun_bc_t.child_table, or in
mp_obj_fun_bc_t.child_table[num_children] if num_children > 0.  The reasons
for doing this are:

1. It decouples the native emitter from runtime requirements, the emitted
   code no longer needs to know if the system it runs on can/can't read
   byte-wise from executable memory.

2. It makes all ports have the same emitter behaviour, there is no longer
   the N_PRELUDE_AS_BYTES_OBJ option.

3. The module's constant table is now used only for actual constants in the
   Python code.  This allows further optimisations to be done with the
   constants (eg constant deduplication).

Code size change for those ports that enable the native emitter:
   unix x64:   +80 +0.015%
      stm32:   +24 +0.004% PYBV10
    esp8266:   +88 +0.013% GENERIC
      esp32:   -20 -0.002% GENERIC[incl -112(data)]
        rp2:   +32 +0.005% PICO

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 16:44:49 +10:00
Damien George 8725a32f41 tools/ci.sh: For esp8266 CI install esptool 3.3.1.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 16:01:56 +10:00
Damien George f8d3956c63 tools/ci.sh: Run full test suite on stackless and settrace builds.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 14:25:51 +10:00
Damien George c49d5207e9 py/persistentcode: Remove unicode feature flag from .mpy file.
Prior to this commit, even with unicode disabled .py and .mpy files could
contain unicode characters, eg by entering them directly in a string as
utf-8 encoded.

The only thing the compiler disallowed (with unicode disabled) was using
\uxxxx and \Uxxxxxxxx notation to specify a character within a string with
value >= 0x100; that would give a SyntaxError.

With this change mpy-cross will now accept \u and \U notation to insert a
character with value >= 0x100 into a string (because the -mno-unicode
option is now gone, there's no way to forbid this).  The runtime will
happily work with strings with such characters, just like it already works
with strings with characters that were utf-8 encoded directly.

This change simplifies things because there are no longer any feature
flags in .mpy files, and any bytecode .mpy will now run on any target.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 12:51:54 +10:00
Damien George 2ed4f7a130 tools/mpy_ld.py: Remove obsolete QSTR_WINDOW_SIZE constant.
This was made obsolete in f2040bfc7e

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 12:51:54 +10:00
Takeo Takahashi e3c880a569 tools/autobuild: Update for ports/renesas-ra.
* Add build_renesas_ra_boards call in autobuild.sh
* Add build_renesas_ra_boards function to generate firmware.hex.

Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
2022-05-06 17:15:21 +09:00
Damien George aec8db3cd1 tools/uncrustify: Make sure a space exists after a comma.
This will add a space after a comma if it doesn't have one, but will allow
more than one space if the spaces are already there.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-05 13:30:40 +10:00
Takeo Takahashi 3a941cce51 tools/ci.sh: Update for ports/renesas-ra.
Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
2022-04-29 12:29:07 +09:00
Damien George 81c9219375 tools/ci.sh: Don't run uasyncio_gather test on macOS.
The timing on macOS makes this test unreliable.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-21 14:25:17 +10:00
Damien George fb10d15d47 tools/codeformat.py: Include more ports .c/.h files in formatting.
This adds a rule to cover all ports/**/*.[ch] file to the code formatting
list.  Explicit exclusions are also added for code in ports/ which is third
party, or which requires a lot of reformatting.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-20 16:02:11 +10:00
Damien George 71a64545a8 tools/codeformat.py: Remove ports/unix/*.py from exclusion list.
These .py files were removed from the .gitignore in commit
3c2aa5ff93

Signed-off-by: Damien George <damien@micropython.org>
2022-04-20 16:01:11 +10:00
Damien George 07f526067e tools/mpy-tool.py: Intern more strings when freezing.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 23:52:14 +10:00
Damien George 40d431d1bb tools/mpy-tool.py: Optimise freezing of str when str data is a qstr.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 23:52:14 +10:00
Damien George e647966fc9 tools/mpy-tool.py: Make global qstr list a dedicated class.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 23:52:14 +10:00
Damien George dfc6c6299c tools/mpy-tool.py: Optimise freezing of empty str and bytes objects.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 23:52:14 +10:00
Damien George 9c8a56343f tools/mpy-tool.py: Optimise freezing of ints that can fit a small int.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 23:52:14 +10:00
Damien George 68b3aeeb57 tools/mpy-tool.py: Support freezing tuples and other consts.
This also simplifies how constants are frozen.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 23:52:14 +10:00
Damien George 2a075cc8a9 tools/mpy-tool.py: Support loading tuples from .mpy files.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 23:52:14 +10:00
Damien George 42d0bd2c17 py/persistentcode: Define enum values for obj types instead of letters.
To keep the separate parts of the code that use these values in sync.  And
make it easier to add new object types.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 22:44:04 +10:00
Damien George be25e333df esp32/boards: Add test manifest to test freezing during build.
Includes tests for freezing @micropython.native and @micropython.viper
code.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 14:06:38 +10:00
Rob Knegjens d3d862769a tools/mpremote: Add option to mount cmd to allow "unsafe" symlinks.
Specifying the option `--unsafe-links` (or `-l`) to `mpremote mount` will
allow symlinks to be followed in the local directory that point outside of
the base directory path.

For the unsafe case the `path_check()` method of `PyboardCommand` still
checks for a common path but without expanding symlinks.  While this check
is currently redundant, it makes the purpose of the method clearer for
possible future uses or extensions.
2022-04-11 15:15:04 +10:00
Rob Knegjens 56978c3dde tools/mpremote: Show progress indicator when copying large files.
When copying large files (> 2048 bytes) to or from a device with
`mpremote cp` a progress bar and percentage counter are temporarily shown.
2022-04-11 15:04:20 +10:00
Damien George b59989f40e rp2/Makefile: Add mbedtls to GIT_SUBMODULES list and use it in CI func.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-11 13:05:25 +10:00
Damien George 86e6744ff5 rp2: Enable ucryptolib, using mbedtls for backend functions.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-04 23:15:01 +10:00
Andrew Leech 79c05bd522 tools/mpremote: Improve reliability of mount after soft reboot.
With the existing code problems can occur with remounting, the "if t -
t_last_activity > QUIET_TIMEOUT:" check can be triggered early before the
REPL string comes through, meaning that the remount doesn't happen.

On certain boards the "MPY: soft reboot" line comes through immediately
(getting the routine past initial timeout) but then there's a slightly
longer delay while the board restarts before it prints out the startup
header and the REPL prompt.

This commit adds some extra pattern monitoring during the timeout loop to
track the state if a soft restart is actually started.
2022-04-04 16:49:38 +10:00
Waterlens 4c252ae067 tools/mpremote: Allow running mpremote with `python -m`.
This is helpful because some scripts are likely to use mpremote with a
specific python path.
2022-03-30 15:29:26 +11:00
Damien George 35dbde163a tools/mpremote: Support any prompt string when detecting soft reset.
The prompt may be changed by sys.ps1.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-28 16:47:50 +11:00