RPi 4 sysupgrade [21.02.1 squashfs overlay retained]

i've just repeated and validated your findings...

specifically;

  • when keep settings is unchecked
  • on official squashfs images (21.02.0>21.02.1)

rootfs_data/overlay filesystem is retained leading to the above symptoms...;

  • note: kernel + mii reported info is purely metadata and does not reflect real packages on /rom / system
  • config files/settings are retained

additionally;

  • same behavior is present upgrading from 21.02.1>21.02.1 (unresolved in 21.02.1)
  • rootfs_data/overlay is also retained when keep settings is selected (in otherwords everyone on 21.02.1(upgraded-to) + squashfs + sdcard/rpi?1 has corrupted/stale opkg metadata)

1

thinking back over the changes on this target... you'll probably find this is a 'c' level issue i.e. block_mount
in which case... it would probably effect more than just the rpi4-squashfs

(if as you state rc3>rc4 or whatever was ok)

(but it seems not many people select 'do not keep settings' + use squashfs)


edit: bug is resolved in master branch

hmmm... they all seem to contain the same fs-tools version so maybe not 'c' after all...

1 Like

late to the party but it's seems to confirm what i could no understand in my other post here.. this was with squashfs trying to no retain settings.. maybe this is useful otherwise sorry for the noise..

1 Like

I encountered exactly the same under exactly the described circumstances, but I too didn't understand what was actually happening at the time. I backup up settings, flashed clean with 21.02.1 and restored the backup and all was well, but overlayfs was definitely in a wonky state after sysupgrade.

1 Like

Shouldn't this buggy upgrade be reported directly to Devs so they can fix source ?
I use EXT4, so I never encountered problem...

1 Like

Thank you for validating my findings and taking a closer look than I could.

I don't know if it helps but I noticed that the bug only occurs after user-installing a package.
So if you sysupgrade a completely untouched (or only setting a password and changing nothing else) 21.02.0>21.02.1 Luci shows the expected versions of the kernel and libwolfssl (though I did not test installing any 5.4.154 packages at that point).
Only after I had installed kmod-mii before the sysupgrade did the bug occur at which point I stopped testing.

Is it only an issue with the metadata? After all the first symptoms I noticed were the eth1 device (usb lan adapter) being present but not working and not being able to install the 5.4.154 packages after the 21.02.1 upgrade.

Is "Perform reset" and setting up from there ok or is it not a valid workaround?

1 Like

not being a squashfs user... i can't really answer that with confidence but in theory;

  • backup> sysupgrate > reset > restore backup
    or
  • backup > remove sdcard > flash factory > boot > restore backup

should both be ok based on the above theory...
(i'd only excluded restoring backups from the tests as it would have complicated things before we knew more about what was happening)


I guess well find out exactly when developers take a look;

afaik;

  • eth1 would not have worked due to no drivers really being present (or thereabouts) (expected)
  • opkg would not have worked due to the metadata telling opkg that different versions of stuff are installed on the system when they are not (expected)

you can further validate these things prior to reset with;

find /overlay/upper
dmesg | grep mount_root

Actually, that's the interesting thing - at least in my case, /overlay was empty after the break, yet my configuration was semi-working...almost like squashfs somehow converted itself into an ext4 deployment through sysupgrade. (Yes, it's as ridiculous as it sounds, but the whole thing is a bit hard to believe until you see it.)

1 Like

yeah... I did not actually look directly at /overlay/upper but used opkg list-installed which was reporting the package versions from the old system...

i did look in /lib/modules and there was no modules directory there for the previous kernel (mii) so that seems to line up with what you are saying...

also on the effected systems / borked upgrades... there is a noticable absence of this in dmesg;

[    5.348648] mount_root: rootdisk overlay filesystem has not been formatted yet

There is a serious concern with squashfs sysupgrade, but what is so wrong going with EXT4 in order to avoid all of these issues ?

Unless Devs can confirm solution is in the process of being fixed...

EXT4 is not optimal for flash storage, however this is much more of an issue with non-removable storage.

Removable sd is easily replaced with inexpensive fresh storage/install in this case with RPi4 in the event of premature ageing due to EXT4 unoptimized flash wear & tear.

Squashfs is key to a long life for my EA/MR8300's as they don't have removable system flash, but what's the real concern here with respect to removable sd ?

bug reported on OP behalf;

https://bugs.openwrt.ifw.cn/index.php?do=details&task_id=4131

2 Likes

Thank you, that's appreciated!

I did that and since you also mentioned /lib/modules in another post I also used find /lib/modules just in case.
This was done at four different points:

  1. After the first boot with a fresh 21.02.0 install.
  2. After installing just the kmod-mii package and another reboot.
  3. After the sysupgrade.
  4. Finally after the reset.

I think kmod-mii was present in /overlay/upper after the sysupgrade but since I don't know not what to look for please check the outputs for yourself:

21.02.0 fresh install after first boot

21.02.0 fresh install after first boot:

root@OpenWrt:~# find /overlay/upper
/overlay/upper
/overlay/upper/etc
/overlay/upper/etc/urandom.seed
/overlay/upper/etc/board.json
/overlay/upper/etc/uhttpd.crt
/overlay/upper/etc/dropbear
/overlay/upper/etc/dropbear/dropbear_ed25519_host_key
/overlay/upper/etc/dropbear/dropbear_rsa_host_key
/overlay/upper/etc/uci-defaults
/overlay/upper/etc/uci-defaults/50-dnsmasq-migrate-resolv-conf-auto.sh
/overlay/upper/etc/uci-defaults/50_luci-mod-admin-full
/overlay/upper/etc/uci-defaults/10_migrate-shadow
/overlay/upper/etc/uci-defaults/20_migrate-feeds
/overlay/upper/etc/uci-defaults/15_odhcpd
/overlay/upper/etc/uci-defaults/12_network-generate-ula
/overlay/upper/etc/uci-defaults/14_migrate-dhcp-release
/overlay/upper/etc/uci-defaults/30_luci-theme-bootstrap
/overlay/upper/etc/uci-defaults/50-migrate-rpcd-ubus-sock.sh
/overlay/upper/etc/uci-defaults/13_fix-group-user
/overlay/upper/etc/uci-defaults/00_uhttpd_ubus
/overlay/upper/etc/uhttpd.key
/overlay/upper/etc/config
/overlay/upper/etc/config/network
/overlay/upper/etc/config/luci
/overlay/upper/etc/config/uhttpd
/overlay/upper/etc/config/ucitrack
/overlay/upper/etc/config/wireless
/overlay/upper/etc/config/firewall
/overlay/upper/etc/config/dropbear
/overlay/upper/etc/config/system
/overlay/upper/etc/config/dhcp
/overlay/upper/etc/config/rpcd
/overlay/upper/boot
root@OpenWrt:~#
root@OpenWrt:~# find /lib/modules
/lib/modules
/lib/modules/5.4.143
/lib/modules/5.4.143/brcmfmac.ko
/lib/modules/5.4.143/brcmutil.ko
/lib/modules/5.4.143/cfg80211.ko
/lib/modules/5.4.143/compat.ko
/lib/modules/5.4.143/crc-ccitt.ko
/lib/modules/5.4.143/evdev.ko
/lib/modules/5.4.143/fat.ko
/lib/modules/5.4.143/hid-generic.ko
/lib/modules/5.4.143/hid.ko
/lib/modules/5.4.143/ip6_tables.ko
/lib/modules/5.4.143/ip6t_REJECT.ko
/lib/modules/5.4.143/ip6table_filter.ko
/lib/modules/5.4.143/ip6table_mangle.ko
/lib/modules/5.4.143/ip_tables.ko
/lib/modules/5.4.143/ipt_REJECT.ko
/lib/modules/5.4.143/iptable_filter.ko
/lib/modules/5.4.143/iptable_mangle.ko
/lib/modules/5.4.143/iptable_nat.ko
/lib/modules/5.4.143/nf_conntrack.ko
/lib/modules/5.4.143/nf_defrag_ipv4.ko
/lib/modules/5.4.143/nf_defrag_ipv6.ko
/lib/modules/5.4.143/nf_flow_table.ko
/lib/modules/5.4.143/nf_flow_table_hw.ko
/lib/modules/5.4.143/nf_log_common.ko
/lib/modules/5.4.143/nf_log_ipv4.ko
/lib/modules/5.4.143/nf_log_ipv6.ko
/lib/modules/5.4.143/nf_nat.ko
/lib/modules/5.4.143/nf_reject_ipv4.ko
/lib/modules/5.4.143/nf_reject_ipv6.ko
/lib/modules/5.4.143/nls_cp437.ko
/lib/modules/5.4.143/nls_iso8859-1.ko
/lib/modules/5.4.143/nls_utf8.ko
/lib/modules/5.4.143/ppp_async.ko
/lib/modules/5.4.143/ppp_generic.ko
/lib/modules/5.4.143/pppoe.ko
/lib/modules/5.4.143/pppox.ko
/lib/modules/5.4.143/slhc.ko
/lib/modules/5.4.143/snd-bcm2835.ko
/lib/modules/5.4.143/snd-compress.ko
/lib/modules/5.4.143/snd-hwdep.ko
/lib/modules/5.4.143/snd-mixer-oss.ko
/lib/modules/5.4.143/snd-pcm-dmaengine.ko
/lib/modules/5.4.143/snd-pcm-oss.ko
/lib/modules/5.4.143/snd-pcm.ko
/lib/modules/5.4.143/snd-rawmidi.ko
/lib/modules/5.4.143/snd-seq-device.ko
/lib/modules/5.4.143/snd-timer.ko
/lib/modules/5.4.143/snd.ko
/lib/modules/5.4.143/soundcore.ko
/lib/modules/5.4.143/usbhid.ko
/lib/modules/5.4.143/vfat.ko
/lib/modules/5.4.143/x_tables.ko
/lib/modules/5.4.143/xt_CT.ko
/lib/modules/5.4.143/xt_FLOWOFFLOAD.ko
/lib/modules/5.4.143/xt_LOG.ko
/lib/modules/5.4.143/xt_MASQUERADE.ko
/lib/modules/5.4.143/xt_REDIRECT.ko
/lib/modules/5.4.143/xt_TCPMSS.ko
/lib/modules/5.4.143/xt_comment.ko
/lib/modules/5.4.143/xt_conntrack.ko
/lib/modules/5.4.143/xt_limit.ko
/lib/modules/5.4.143/xt_mac.ko
/lib/modules/5.4.143/xt_mark.ko
/lib/modules/5.4.143/xt_multiport.ko
/lib/modules/5.4.143/xt_nat.ko
/lib/modules/5.4.143/xt_state.ko
/lib/modules/5.4.143/xt_tcpudp.ko
/lib/modules/5.4.143/xt_time.ko
root@OpenWrt:~#
root@OpenWrt:~# dmesg | grep -i "mount_root" -C 3
[    2.218708] bcmgenet: Skipping UMAC reset
[    2.224079] bcmgenet fd580000.ethernet: configuring instance for external RGMII
[    2.231578] bcmgenet fd580000.ethernet eth0: Link is Down
[    6.320916] mount_root: rootdisk overlay filesystem has not been formatted yet
[    7.363901] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[    7.372151] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    7.683247] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[    7.691257] mount_root: overlay filesystem has not been fully initialized yet
[    7.698930] mount_root: switching to ext4 overlay
[    7.707330] overlayfs: "xino" feature enabled using 32 upper inode bits.
[    7.744351] urandom-seed: Seed file not found (/etc/urandom.seed)
[    7.805805] bcmgenet fd580000.ethernet eth0: Link is Down
root@OpenWrt:~#
21.02.0 fresh install after kmod-mii installation

21.02.0 fresh install after kmod-mii installation:

root@OpenWrt:~# find /overlay/upper
/overlay/upper
/overlay/upper/etc
/overlay/upper/etc/modules.d
/overlay/upper/etc/modules.d/15-mii
/overlay/upper/etc/urandom.seed
/overlay/upper/etc/board.json
/overlay/upper/etc/uhttpd.crt
/overlay/upper/etc/dropbear
/overlay/upper/etc/dropbear/dropbear_ed25519_host_key
/overlay/upper/etc/dropbear/dropbear_rsa_host_key
/overlay/upper/etc/uci-defaults
/overlay/upper/etc/uci-defaults/50-dnsmasq-migrate-resolv-conf-auto.sh
/overlay/upper/etc/uci-defaults/50_luci-mod-admin-full
/overlay/upper/etc/uci-defaults/10_migrate-shadow
/overlay/upper/etc/uci-defaults/20_migrate-feeds
/overlay/upper/etc/uci-defaults/15_odhcpd
/overlay/upper/etc/uci-defaults/12_network-generate-ula
/overlay/upper/etc/uci-defaults/14_migrate-dhcp-release
/overlay/upper/etc/uci-defaults/30_luci-theme-bootstrap
/overlay/upper/etc/uci-defaults/50-migrate-rpcd-ubus-sock.sh
/overlay/upper/etc/uci-defaults/13_fix-group-user
/overlay/upper/etc/uci-defaults/00_uhttpd_ubus
/overlay/upper/etc/uhttpd.key
/overlay/upper/etc/modules-boot.d
/overlay/upper/etc/modules-boot.d/15-mii
/overlay/upper/etc/config
/overlay/upper/etc/config/network
/overlay/upper/etc/config/luci
/overlay/upper/etc/config/uhttpd
/overlay/upper/etc/config/ucitrack
/overlay/upper/etc/config/wireless
/overlay/upper/etc/config/firewall
/overlay/upper/etc/config/dropbear
/overlay/upper/etc/config/system
/overlay/upper/etc/config/dhcp
/overlay/upper/etc/config/rpcd
/overlay/upper/usr
/overlay/upper/usr/lib
/overlay/upper/usr/lib/opkg
/overlay/upper/usr/lib/opkg/info
/overlay/upper/usr/lib/opkg/info/kmod-mii.control
/overlay/upper/usr/lib/opkg/info/kmod-mii.postinst
/overlay/upper/usr/lib/opkg/info/kmod-mii.list
/overlay/upper/usr/lib/opkg/info/kmod-mii.prerm
/overlay/upper/usr/lib/opkg/status
/overlay/upper/boot
root@OpenWrt:~#
root@OpenWrt:~# find /lib/modules
/lib/modules
/lib/modules/5.4.143
/lib/modules/5.4.143/brcmfmac.ko
/lib/modules/5.4.143/brcmutil.ko
/lib/modules/5.4.143/cfg80211.ko
/lib/modules/5.4.143/compat.ko
/lib/modules/5.4.143/crc-ccitt.ko
/lib/modules/5.4.143/evdev.ko
/lib/modules/5.4.143/fat.ko
/lib/modules/5.4.143/hid-generic.ko
/lib/modules/5.4.143/hid.ko
/lib/modules/5.4.143/ip6_tables.ko
/lib/modules/5.4.143/ip6t_REJECT.ko
/lib/modules/5.4.143/ip6table_filter.ko
/lib/modules/5.4.143/ip6table_mangle.ko
/lib/modules/5.4.143/ip_tables.ko
/lib/modules/5.4.143/ipt_REJECT.ko
/lib/modules/5.4.143/iptable_filter.ko
/lib/modules/5.4.143/iptable_mangle.ko
/lib/modules/5.4.143/iptable_nat.ko
/lib/modules/5.4.143/nf_conntrack.ko
/lib/modules/5.4.143/nf_defrag_ipv4.ko
/lib/modules/5.4.143/nf_defrag_ipv6.ko
/lib/modules/5.4.143/nf_flow_table.ko
/lib/modules/5.4.143/nf_flow_table_hw.ko
/lib/modules/5.4.143/nf_log_common.ko
/lib/modules/5.4.143/nf_log_ipv4.ko
/lib/modules/5.4.143/nf_log_ipv6.ko
/lib/modules/5.4.143/nf_nat.ko
/lib/modules/5.4.143/nf_reject_ipv4.ko
/lib/modules/5.4.143/nf_reject_ipv6.ko
/lib/modules/5.4.143/nls_cp437.ko
/lib/modules/5.4.143/nls_iso8859-1.ko
/lib/modules/5.4.143/nls_utf8.ko
/lib/modules/5.4.143/ppp_async.ko
/lib/modules/5.4.143/ppp_generic.ko
/lib/modules/5.4.143/pppoe.ko
/lib/modules/5.4.143/pppox.ko
/lib/modules/5.4.143/slhc.ko
/lib/modules/5.4.143/snd-bcm2835.ko
/lib/modules/5.4.143/snd-compress.ko
/lib/modules/5.4.143/snd-hwdep.ko
/lib/modules/5.4.143/snd-mixer-oss.ko
/lib/modules/5.4.143/snd-pcm-dmaengine.ko
/lib/modules/5.4.143/snd-pcm-oss.ko
/lib/modules/5.4.143/snd-pcm.ko
/lib/modules/5.4.143/snd-rawmidi.ko
/lib/modules/5.4.143/snd-seq-device.ko
/lib/modules/5.4.143/snd-timer.ko
/lib/modules/5.4.143/snd.ko
/lib/modules/5.4.143/soundcore.ko
/lib/modules/5.4.143/usbhid.ko
/lib/modules/5.4.143/vfat.ko
/lib/modules/5.4.143/x_tables.ko
/lib/modules/5.4.143/xt_CT.ko
/lib/modules/5.4.143/xt_FLOWOFFLOAD.ko
/lib/modules/5.4.143/xt_LOG.ko
/lib/modules/5.4.143/xt_MASQUERADE.ko
/lib/modules/5.4.143/xt_REDIRECT.ko
/lib/modules/5.4.143/xt_TCPMSS.ko
/lib/modules/5.4.143/xt_comment.ko
/lib/modules/5.4.143/xt_conntrack.ko
/lib/modules/5.4.143/xt_limit.ko
/lib/modules/5.4.143/xt_mac.ko
/lib/modules/5.4.143/xt_mark.ko
/lib/modules/5.4.143/xt_multiport.ko
/lib/modules/5.4.143/xt_nat.ko
/lib/modules/5.4.143/xt_state.ko
/lib/modules/5.4.143/xt_tcpudp.ko
/lib/modules/5.4.143/xt_time.ko
root@OpenWrt:~#
root@OpenWrt:~# dmesg | grep -i "mount_root" -C 3
[    2.202955] bcmgenet fd580000.ethernet eth0: Link is Down
[    6.414169] EXT4-fs (loop0): recovery complete
[    6.420540] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[    6.428615] mount_root: switching to ext4 overlay
[    6.434561] overlayfs: "xino" feature enabled using 32 upper inode bits.
[    6.474368] urandom-seed: Seeding with /etc/urandom.seed
[    6.554075] procd: - early -
root@OpenWrt:~#
21.02.1 after sysupgrade

21.02.1 after sysupgrade:

root@OpenWrt:~# find /overlay/upper
/overlay/upper
/overlay/upper/etc
/overlay/upper/etc/modules.d
/overlay/upper/etc/modules.d/15-mii
/overlay/upper/etc/urandom.seed
/overlay/upper/etc/board.json
/overlay/upper/etc/uhttpd.crt
/overlay/upper/etc/dropbear
/overlay/upper/etc/dropbear/dropbear_ed25519_host_key
/overlay/upper/etc/dropbear/dropbear_rsa_host_key
/overlay/upper/etc/uci-defaults
/overlay/upper/etc/uci-defaults/50-dnsmasq-migrate-resolv-conf-auto.sh
/overlay/upper/etc/uci-defaults/50_luci-mod-admin-full
/overlay/upper/etc/uci-defaults/10_migrate-shadow
/overlay/upper/etc/uci-defaults/20_migrate-feeds
/overlay/upper/etc/uci-defaults/15_odhcpd
/overlay/upper/etc/uci-defaults/12_network-generate-ula
/overlay/upper/etc/uci-defaults/14_migrate-dhcp-release
/overlay/upper/etc/uci-defaults/30_luci-theme-bootstrap
/overlay/upper/etc/uci-defaults/50-migrate-rpcd-ubus-sock.sh
/overlay/upper/etc/uci-defaults/13_fix-group-user
/overlay/upper/etc/uci-defaults/00_uhttpd_ubus
/overlay/upper/etc/uhttpd.key
/overlay/upper/etc/modules-boot.d
/overlay/upper/etc/modules-boot.d/15-mii
/overlay/upper/etc/config
/overlay/upper/etc/config/network
/overlay/upper/etc/config/luci
/overlay/upper/etc/config/uhttpd
/overlay/upper/etc/config/ucitrack
/overlay/upper/etc/config/wireless
/overlay/upper/etc/config/firewall
/overlay/upper/etc/config/dropbear
/overlay/upper/etc/config/system
/overlay/upper/etc/config/dhcp
/overlay/upper/etc/config/rpcd
/overlay/upper/usr
/overlay/upper/usr/lib
/overlay/upper/usr/lib/opkg
/overlay/upper/usr/lib/opkg/info
/overlay/upper/usr/lib/opkg/info/kmod-mii.control
/overlay/upper/usr/lib/opkg/info/kmod-mii.postinst
/overlay/upper/usr/lib/opkg/info/kmod-mii.list
/overlay/upper/usr/lib/opkg/info/kmod-mii.prerm
/overlay/upper/usr/lib/opkg/status
/overlay/upper/boot
root@OpenWrt:~#
root@OpenWrt:~# find /lib/modules
/lib/modules
/lib/modules/5.4.154
/lib/modules/5.4.154/brcmfmac.ko
/lib/modules/5.4.154/brcmutil.ko
/lib/modules/5.4.154/cfg80211.ko
/lib/modules/5.4.154/compat.ko
/lib/modules/5.4.154/crc-ccitt.ko
/lib/modules/5.4.154/evdev.ko
/lib/modules/5.4.154/fat.ko
/lib/modules/5.4.154/hid-generic.ko
/lib/modules/5.4.154/hid.ko
/lib/modules/5.4.154/ip6_tables.ko
/lib/modules/5.4.154/ip6t_REJECT.ko
/lib/modules/5.4.154/ip6table_filter.ko
/lib/modules/5.4.154/ip6table_mangle.ko
/lib/modules/5.4.154/ip_tables.ko
/lib/modules/5.4.154/ipt_REJECT.ko
/lib/modules/5.4.154/iptable_filter.ko
/lib/modules/5.4.154/iptable_mangle.ko
/lib/modules/5.4.154/iptable_nat.ko
/lib/modules/5.4.154/nf_conntrack.ko
/lib/modules/5.4.154/nf_defrag_ipv4.ko
/lib/modules/5.4.154/nf_defrag_ipv6.ko
/lib/modules/5.4.154/nf_flow_table.ko
/lib/modules/5.4.154/nf_flow_table_hw.ko
/lib/modules/5.4.154/nf_log_common.ko
/lib/modules/5.4.154/nf_log_ipv4.ko
/lib/modules/5.4.154/nf_log_ipv6.ko
/lib/modules/5.4.154/nf_nat.ko
/lib/modules/5.4.154/nf_reject_ipv4.ko
/lib/modules/5.4.154/nf_reject_ipv6.ko
/lib/modules/5.4.154/nls_cp437.ko
/lib/modules/5.4.154/nls_iso8859-1.ko
/lib/modules/5.4.154/nls_utf8.ko
/lib/modules/5.4.154/ppp_async.ko
/lib/modules/5.4.154/ppp_generic.ko
/lib/modules/5.4.154/pppoe.ko
/lib/modules/5.4.154/pppox.ko
/lib/modules/5.4.154/slhc.ko
/lib/modules/5.4.154/snd-bcm2835.ko
/lib/modules/5.4.154/snd-compress.ko
/lib/modules/5.4.154/snd-hwdep.ko
/lib/modules/5.4.154/snd-mixer-oss.ko
/lib/modules/5.4.154/snd-pcm-dmaengine.ko
/lib/modules/5.4.154/snd-pcm-oss.ko
/lib/modules/5.4.154/snd-pcm.ko
/lib/modules/5.4.154/snd-rawmidi.ko
/lib/modules/5.4.154/snd-seq-device.ko
/lib/modules/5.4.154/snd-timer.ko
/lib/modules/5.4.154/snd.ko
/lib/modules/5.4.154/soundcore.ko
/lib/modules/5.4.154/usbhid.ko
/lib/modules/5.4.154/vfat.ko
/lib/modules/5.4.154/x_tables.ko
/lib/modules/5.4.154/xt_CT.ko
/lib/modules/5.4.154/xt_FLOWOFFLOAD.ko
/lib/modules/5.4.154/xt_LOG.ko
/lib/modules/5.4.154/xt_MASQUERADE.ko
/lib/modules/5.4.154/xt_REDIRECT.ko
/lib/modules/5.4.154/xt_TCPMSS.ko
/lib/modules/5.4.154/xt_comment.ko
/lib/modules/5.4.154/xt_conntrack.ko
/lib/modules/5.4.154/xt_limit.ko
/lib/modules/5.4.154/xt_mac.ko
/lib/modules/5.4.154/xt_mark.ko
/lib/modules/5.4.154/xt_multiport.ko
/lib/modules/5.4.154/xt_nat.ko
/lib/modules/5.4.154/xt_state.ko
/lib/modules/5.4.154/xt_tcpudp.ko
/lib/modules/5.4.154/xt_time.ko
root@OpenWrt:~#
root@OpenWrt:~# dmesg | grep -i "mount_root" -C 3
[    6.307982] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[    6.316218] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    6.326167] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[    6.336391] mount_root: switching to ext4 overlay
[    6.347177] overlayfs: "xino" feature enabled using 32 upper inode bits.
[    6.388946] urandom-seed: Seeding with /etc/urandom.seed
[    6.453887] bcmgenet fd580000.ethernet eth0: Link is Down
root@OpenWrt:~#
21.02.1 after reset

21.02.1 after reset

root@OpenWrt:~# find /overlay/upper
/overlay/upper
/overlay/upper/etc
/overlay/upper/etc/urandom.seed
/overlay/upper/etc/board.json
/overlay/upper/etc/uhttpd.crt
/overlay/upper/etc/dropbear
/overlay/upper/etc/dropbear/dropbear_ed25519_host_key
/overlay/upper/etc/dropbear/dropbear_rsa_host_key
/overlay/upper/etc/uci-defaults
/overlay/upper/etc/uci-defaults/50-dnsmasq-migrate-resolv-conf-auto.sh
/overlay/upper/etc/uci-defaults/50_luci-mod-admin-full
/overlay/upper/etc/uci-defaults/10_migrate-shadow
/overlay/upper/etc/uci-defaults/20_migrate-feeds
/overlay/upper/etc/uci-defaults/15_odhcpd
/overlay/upper/etc/uci-defaults/12_network-generate-ula
/overlay/upper/etc/uci-defaults/14_migrate-dhcp-release
/overlay/upper/etc/uci-defaults/30_luci-theme-bootstrap
/overlay/upper/etc/uci-defaults/50-migrate-rpcd-ubus-sock.sh
/overlay/upper/etc/uci-defaults/13_fix-group-user
/overlay/upper/etc/uci-defaults/00_uhttpd_ubus
/overlay/upper/etc/uhttpd.key
/overlay/upper/etc/config
/overlay/upper/etc/config/network
/overlay/upper/etc/config/luci
/overlay/upper/etc/config/uhttpd
/overlay/upper/etc/config/ucitrack
/overlay/upper/etc/config/wireless
/overlay/upper/etc/config/firewall
/overlay/upper/etc/config/dropbear
/overlay/upper/etc/config/system
/overlay/upper/etc/config/dhcp
/overlay/upper/etc/config/rpcd
/overlay/upper/boot
root@OpenWrt:~#
root@OpenWrt:~# find /lib/modules
/lib/modules
/lib/modules/5.4.154
/lib/modules/5.4.154/brcmfmac.ko
/lib/modules/5.4.154/brcmutil.ko
/lib/modules/5.4.154/cfg80211.ko
/lib/modules/5.4.154/compat.ko
/lib/modules/5.4.154/crc-ccitt.ko
/lib/modules/5.4.154/evdev.ko
/lib/modules/5.4.154/fat.ko
/lib/modules/5.4.154/hid-generic.ko
/lib/modules/5.4.154/hid.ko
/lib/modules/5.4.154/ip6_tables.ko
/lib/modules/5.4.154/ip6t_REJECT.ko
/lib/modules/5.4.154/ip6table_filter.ko
/lib/modules/5.4.154/ip6table_mangle.ko
/lib/modules/5.4.154/ip_tables.ko
/lib/modules/5.4.154/ipt_REJECT.ko
/lib/modules/5.4.154/iptable_filter.ko
/lib/modules/5.4.154/iptable_mangle.ko
/lib/modules/5.4.154/iptable_nat.ko
/lib/modules/5.4.154/nf_conntrack.ko
/lib/modules/5.4.154/nf_defrag_ipv4.ko
/lib/modules/5.4.154/nf_defrag_ipv6.ko
/lib/modules/5.4.154/nf_flow_table.ko
/lib/modules/5.4.154/nf_flow_table_hw.ko
/lib/modules/5.4.154/nf_log_common.ko
/lib/modules/5.4.154/nf_log_ipv4.ko
/lib/modules/5.4.154/nf_log_ipv6.ko
/lib/modules/5.4.154/nf_nat.ko
/lib/modules/5.4.154/nf_reject_ipv4.ko
/lib/modules/5.4.154/nf_reject_ipv6.ko
/lib/modules/5.4.154/nls_cp437.ko
/lib/modules/5.4.154/nls_iso8859-1.ko
/lib/modules/5.4.154/nls_utf8.ko
/lib/modules/5.4.154/ppp_async.ko
/lib/modules/5.4.154/ppp_generic.ko
/lib/modules/5.4.154/pppoe.ko
/lib/modules/5.4.154/pppox.ko
/lib/modules/5.4.154/slhc.ko
/lib/modules/5.4.154/snd-bcm2835.ko
/lib/modules/5.4.154/snd-compress.ko
/lib/modules/5.4.154/snd-hwdep.ko
/lib/modules/5.4.154/snd-mixer-oss.ko
/lib/modules/5.4.154/snd-pcm-dmaengine.ko
/lib/modules/5.4.154/snd-pcm-oss.ko
/lib/modules/5.4.154/snd-pcm.ko
/lib/modules/5.4.154/snd-rawmidi.ko
/lib/modules/5.4.154/snd-seq-device.ko
/lib/modules/5.4.154/snd-timer.ko
/lib/modules/5.4.154/snd.ko
/lib/modules/5.4.154/soundcore.ko
/lib/modules/5.4.154/usbhid.ko
/lib/modules/5.4.154/vfat.ko
/lib/modules/5.4.154/x_tables.ko
/lib/modules/5.4.154/xt_CT.ko
/lib/modules/5.4.154/xt_FLOWOFFLOAD.ko
/lib/modules/5.4.154/xt_LOG.ko
/lib/modules/5.4.154/xt_MASQUERADE.ko
/lib/modules/5.4.154/xt_REDIRECT.ko
/lib/modules/5.4.154/xt_TCPMSS.ko
/lib/modules/5.4.154/xt_comment.ko
/lib/modules/5.4.154/xt_conntrack.ko
/lib/modules/5.4.154/xt_limit.ko
/lib/modules/5.4.154/xt_mac.ko
/lib/modules/5.4.154/xt_mark.ko
/lib/modules/5.4.154/xt_multiport.ko
/lib/modules/5.4.154/xt_nat.ko
/lib/modules/5.4.154/xt_state.ko
/lib/modules/5.4.154/xt_tcpudp.ko
/lib/modules/5.4.154/xt_time.ko
root@OpenWrt:~#
root@OpenWrt:~# dmesg | grep -i "mount_root" -C 3
[    6.378193] EXT4-fs (loop0): 6 orphan inodes deleted
[    6.383186] EXT4-fs (loop0): recovery complete
[    6.404678] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[    6.413026] mount_root: overlay filesystem has not been fully initialized yet
[    6.420421] mount_root: switching to ext4 overlay
[    6.426334] overlayfs: "xino" feature enabled using 32 upper inode bits.
[    6.461169] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[    6.472320] urandom-seed: Seed file not found (/etc/urandom.seed)
root@OpenWrt:~#
1 Like

yes... in output (3) after not keeping settings (or even with keeping settings) the above should not be present after a sysupgrade...

the stale opkg/status file is the one that's wreaking havoc with opkg install and kernel version mistmatch...


when you 'reset' ... it's just wiping all this...

Short update:
Not sure whether the issue stems from this bug or something I did here but at some point the system started losing setting changes, additionally installed packages vanished and uninstalled packages reappeared after a reboot.
So I flashed a stock ext4 21.02.1 factory image on the spare SD card and won't do fancy things anymore (for now).

my sincere apologies... it seems that bug tracking system was abandoned and there is conflicting information about whether existing reports will be migrated so someone may have to re-lodge this bug if you want it fixed

A migration of existing issues is not planed

1 Like

Hi,

you helped a lot in verifying the issue, reported it in the old system in my stead and now three months later notified me about the move to GitHub I would otherwise probably have missed.
So you went above and beyond and there is absolutely nothing for you to apologize for!

If that's ok with you I plan on opening an issue at GitHub in which I'd just link to the issue you opened and copy your description and the comments of the bug.

sounds good... thankyou very much

Done (hopefully done correctly, first time I used GitHub).

2 Likes

no change for 21.02.2

1 Like

Thank you for the heads up.

1 Like

FYI: Just did a sysupgrade from 21.02.2 > 21.02.3 with the official squashfs image, this time "keep settings" was checked.
Could not reproduce the bug!

Was there a fix implemented? No mention of any commit in the GitHub issue.