Prior to this patch, when a lot of data was output by a running script
pyboard.py would try to capture all of this output into the "data"
variable, which would gradually slow down pyboard.py to the point where it
would have large CPU and memory usage (on the host) and potentially lose
data.
This patch fixes this problem by not accumulating the data in the case that
the data is not needed, which is when "data_consumer" is used.
The qstr window size is not log-2 encoded, it's just the actual number (but
in mpy-tool.py this didn't lead to an error because the size is just used
to truncate the window so it doesn't grow arbitrarily large in memory).
Addresses issue #4635.
This system makes it a lot easier to include external libraries as static,
native modules in MicroPython. Simply pass USER_C_MODULES (like
FROZEN_MPY_DIR) as a make parameter.
When encoded in the mpy file, if qstr <= QSTR_LAST_STATIC then store two
bytes: 0, static_qstr_id. Otherwise encode the qstr as usual (either with
string data or a reference into the qstr window).
Reduces mpy file size by about 5%.
Instead of emitting two bytes in the bytecode for where the linked qstr
should be written to, it is now replaced by the actual qstr data, or a
reference into the qstr window.
Reduces mpy file size by about 10%.
This is an implementation of a sliding qstr window used to reduce the
number of qstrs stored in a .mpy file. The window size is configured to 32
entries which takes a fixed 64 bytes (16-bits each) on the C stack when
loading/saving a .mpy file. It allows to remember the most recent 32 qstrs
so they don't need to be stored again in the .mpy file. The qstr window
uses a simple least-recently-used mechanism to discard the least recently
used qstr when the window overflows (similar to dictionary compression).
This scheme only needs a single pass to save/load the .mpy file.
Reduces mpy file size by about 25% with a window size of 32.
POP_BLOCK and POP_EXCEPT are now the same, and are always followed by a
JUMP. So this optimisation reduces code size, and RAM usage of bytecode by
two bytes for each try-except handler.
Under python3 (tested with 3.6.7) bytes with a list of integers as an
argument returns a different result than under python 2.7 (tested with
2.7.15rc1) which causes pydfu.py to fail when run under 2.7. Changing
bytes to bytearray makes pydfu work properly under both Python 2.7 and
Python 3.6.
If you happen to only have a really simple frozen file that doesn't contain
any new qstrs then the generated frozen_mpy.c file contains an empty
enumeration which causes a C compile time error.
Following an equivalent fix to py/bc.c. The reason the incorrect values
for the opcode constants were not previously causing a bug is because they
were never being used: these opcodes always have qstr arguments so the part
of the code that was comparing them would never be reached.
Thanks to @malinah for finding the problem and providing the initial patch.
A DFU device must be in the idle state before it can be programmed, and
this requires either clearing the status or aborting, depending on its
current state. Code is added to do this. And the USB transfer size is now
automatically detected so devices with a size less than 2048 bytes work
correctly.
Some Python linters don't like unconditional except clauses because they
catch SystemExit and KeyboardInterrupt, which usually is not the intended
behaviour.
There appears to be an issue on Windows with CPython >= 3.6,
sys.stdout.flush() raises an exception:
OSError: [WinError 87] The parameter is incorrect
It works fine to just catch and ignore the error on the flush line. Tested
on Windows 10 x64 1803 (Build 17134.228), Python 3.6.4 amd64.
The Python documentation recommends to pass the command as a string when
using Popen(..., shell=True). This is because "sh -c <string>" is used to
execute the command and additional arguments after the command string are
passed to the shell itself (not the executing command).
https://docs.python.org/3.5/library/subprocess.html#subprocess.Popen
The first dynamic qstr pool is double the size of the 'alloc' field of
the last const qstr pool. The built in const qstr pool
(mp_qstr_const_pool) has a hardcoded alloc size of 10, meaning that the
first dynamic pool is allocated space for 20 entries. The alloc size
must be less than or equal to the actual number of qstrs in the pool
(the 'len' field) to ensure that the first dynamically created qstr
triggers the creation of a new pool.
When modules are frozen a second const pool is created (generally
mp_qstr_frozen_const_pool) and linked to the built in pool. However,
this second const pool had its 'alloc' field set to the number of qstrs
in the pool. When freezing a large quantity of modules this can result
in thousands of qstrs being in the pool. This means that the first
dynamically created qstr results in a massive allocation. This commit
sets the alloc size of the frozen qstr pool to 10 or less (if the number
of qstrs in the pool is less than 10). The result of this is that the
allocation behaviour when a dynamic qstr is created is identical with an
without frozen code.
Note that there is the potential for a slight memory inefficiency if the
frozen modules have less than 10 qstrs, as the first few dynamic
allocations will have quite a large overhead, but the geometric growth
soon deals with this.
The ST DFU bootloader supports a transfer size up to 2048 bytes, so send
that much data on each download (to device) packet. This almost halves
total download time.
Instead of passing thru more and more options from tinytest-codegen to
run-tests --list-tests, pipe output of run-tests --list-tests into
tinytest-codegen.
Gets passed to run-tests --list-tests to get actual list of tests to use.
If --target= is not given, legacy set hardcoded in tinytest-codegen itself
is used.
Also, get rid of tinytest test groups - they aren't really used for
anything, and only complicate processing. Besides, one of the next
step is to limit number of tests per a generated file to control
the binary size, which also will require "flat" list of tests.
The way tinytest was used in qemu-arm test target is that it didn't test
much. MicroPython tests are based on matching the test output against
reference output, but qemu-arm's implementation didn't do that, it
effectively tested just that there was no exception during test
execution. "upytesthelper" wrapper was introduce to fix it, and so
test generator is now switched to generate test code for it.
Also, fix PEP8 and other codestyle issues.
This patch allows the following code to run without allocating on the heap:
super().foo(...)
Before this patch such a call would allocate a super object on the heap and
then load the foo method and call it right away. The super object is only
needed to perform the lookup of the method and not needed after that. This
patch makes an optimisation to allocate the super object on the C stack and
discard it right after use.
Changes in code size due to this patch are:
bare-arm: +128
minimal: +232
unix x64: +416
unix nanbox: +364
stmhal: +184
esp8266: +340
cc3200: +128
This allows to execute a command and communicate with its stdin/stdout
via pipes ("exec") or with command-created pseudo-terminal ("execpty"),
to emulate serial access. Immediate usecase is controlling a QEMU process
which emulates board's serial via normal console, but it could be used
e.g. with helper binaries to access real board over other hadware
protocols, etc.
An example of device specification for these cases is:
--device exec:../zephyr/qemu.sh
--device execpty:../zephyr/qemu2.sh
Where qemu.sh contains long-long qemu startup line, or calls another
command. There's a special support in this patch for running the command
in a new terminal session, to support shell wrappers like that (without
new terminal session, only wrapper script would be terminated, but its
child processes would continue to run).
This patch introduces the a small framework to track differences between
uPy and CPython. The framework consists of:
- A set of "tests" which test for an individual feature that differs between
uPy and CPy. Each test is like a normal uPy test in the test suite, but
has a special comment at the start with some meta-data: a category (eg
syntax, core language), a human-readable description of the difference, a
cause, and a workaround. Following the meta-data there is a short code
snippet which demonstrates the difference. See tests/cpydiff directory
for the initial set of tests.
- A program (this patch) which runs all the tests (on uPy and CPy) and
generates nicely-formated .rst documenting the differences.
- Integration into the docs build so that everything is automatic, and the
differences appear in a way that is easy for users to read/reference (see
latter commits).
The idea with using this new framework is:
- When a new difference is found it's easy to write a short test for it,
along with a description, and add it to the existing ones. It's also easy
for contributors to submit tests for differences they find.
- When something is no longer different the tool will give an error and
difference can be removed (or promoted to a proper feature test).
Previous to this patch the qemu-arm tests were compiled with is_relp=true
meaning that the __repl_print__ function was called for all lines of code
in the outer scope. This is not the right behaviour for scripts that are
executed as though they were a file (eg tests).
With this fix the micropython/heapalloc_str.py test now works so it is
removed from the test blacklist.
With caching of map lookups in the bytecode, frozen bytecode can still
work but must be stored in RAM, not ROM. This patch allows mpy-tool.py to
generate code that works with this optimisation, but it's not recommended
to use it on embedded targets (because of lack of RAM).
Previous to this patch pyboard.py would open a new serial connection to
the target for each script that was run, and for any command that was run.
Apart from being inefficient, this meant that the board was soft-reset
between scripts/commands, which precludes scripts from accessing variables
set in a previous one.
This patch changes the behaviour of pyboard.py so that the connection to
the target is created only once, and it's not reset between scripts or any
command that is sent with the -c option.
To make its inclusion as frozen modules in multiple ports less magic.
Ports are just expected to symlink 2 files into their scripts/modules
subdirs.
Unix port updated to use this and in general follow frozen modules setup
tested and tried on baremetal ports, where there's "scripts" predefined
dir (overridable with FROZEN_DIR make var), and a user just drops Python
files there.
This helps to test floating point code on Cortex-M hardware.
As part of this patch the link-time-optimisation was disabled because it
wasn't compatible with software FP support. In particular, the linker
could not find the __aeabi_f2d, __aeabi_d2f etc functions even though they
were provided by lib/libm/math.c.
Frozen modules are now stored with extensions and with '/' as path
separator. In other words, frozen modules paths stored as they are
in normal filesystem.
When an mpy file is frozen it must know the values of certain
configuration variables. This patch provides an explicit check in the
generated C file that the configuration variables are what they are
supposed to be.
The config variable MICROPY_MODULE_FROZEN is now made of two separate
parts: MICROPY_MODULE_FROZEN_STR and MICROPY_MODULE_FROZEN_MPY. This
allows to have none, either or both of frozen strings and frozen mpy
files (aka frozen bytecode).
This fix adds PIDs 9801 and 9802 to the pybcdc.inf file.
When in CDC only mode, it presents itself as a Communcations
device rather than as a composite device. Presenting as a
composite device with only the CDC interface seems to confuse
windows.
To test and make sure that the correct pybcdc.inf was being used,
I used USBDeview from http://www.nirsoft.net/utils/usb_devices_view.html
to uninstall any old pyboard drivers (Use Control-F and search
for pyboard). I found running USBDeview as administrator worked best.
Installing the driver in CDC+MSC mode first is recommended (since the
pybcdc.inf file in on the internal flash drive). Then when you switch
modes everything seems to work properly.
I used https://github.com/dhylands/upy-examples/blob/master/boot_switch.py
to easily switch the pyboard between the various USB modes for testing.
The adapter class "TelnetToSerial" is used to access the Telnet
connection using the same API as with the serial connection. The
function pyboard.run-test() has been removed to made the module
generic and because this small test is no longer needed.
When looking for chars to indicate raw repl is active, look for the full
string of chars to improve reliability of entering raw repl correctly.
Previous to this patch there was the possibility that raw repl was
entered in a dirty state, where not all input chars from previous
invocation were drained.
In raw REPL ">" indicates the prompt. We originally read this character
upon entering the raw REPL, and after reading the last bit of the
output. This patch changes the logic so the ">" is read only just
before trying to send the next command. To make this work (and as an
added feature) the input buffer is now flushed upon entering raw REPL.
The main reason for this change is so that pyboard.py recognises the EOF
when sys.exit() is called on the pyboard. Ie, if you run pyboard.py
with a script that calls sys.exit(), then pyboard.py will exit after
the sys.exit() is called.
Damien George
rhubarbdog
Andrew Leech
Ayke van Laethem
Dave Hylands
Martin Dybdal
Rich Barlow
roland
Keith Wiley
Paul Sokolovsky
stijn
Alexander Steffen
Ville Skyttä
Krzysztof Blazewicz
Rami Ali
Alex March
Peter Hinch
Tom Soulanille
Daniel Campora