Python 3.8.x problems in 20.04

Question

I was already on my way to reverting to a backup image of my MicroSD that was ~1.5 week old until after writing it to a brand new SanDisk Micro SD card, the ext4 filesystem gave a bunch of inode errors during boot time, everything in the OS was more or less totally broken and fsck gave basically an insane list of things that were broken on a filesystem level. It's a brand new Sandisk Micro SD card, but apparently during the backup process, data corruption slipped in through the back door.

I'm running Ubuntu 20.04 on a Raspberry Pi 4B, which means that it's running on arm64 architecture, so suggestions to download the .deb packages straight from kernel.org, or to install i.e. python3-minimal doesn't work, since there aren't any installation candidates for ARM64.

Solutions pointing to .deb and other installation packages that simply do not exist for arm64 architecture at all.
There was also plenty of outdated or downright bad advice altogether -- also, none of the system restore methods described in either the manuals or within Ubuntu's user community (boot CD on a Raspberry Pi?) don't apply to 20.04 on the Raspberry Pi 4B, at least not yet.

My install is still staggering along, but the apt/dpkg update system remains pretty much broken no matter what I try. I have unsuccessfully tried downloading the .deb packages and installing them separately.

As for this moment; Py3compile and Pyclean are not found anywhere and those two are the main culprits in dpkg/apt not working, so this answer doesn't work.

$ sudo apt install python3-all
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-all is already the newest version (3.8.2-0ubuntu2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
4 not fully installed or removed.
Need to get 0 B/129 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up pychess (0.12.2-1build1) ...
/var/lib/dpkg/info/pychess.postinst: 6: pycompile: not found
dpkg: error processing package pychess (--configure):
 installed pychess package post-installation script subprocess returned error ex                                   it status 127
dpkg: error processing package python-pkg-resources (--configure):
 package is in a very bad inconsistent state; you should
 reinstall it before attempting configuration
Setting up gir1.2-dee-1.0 (1.2.7+17.10.20170616-4ubuntu6) ...
/var/lib/dpkg/info/gir1.2-dee-1.0.postinst: 6: pycompile: not found
dpkg: error processing package gir1.2-dee-1.0 (--configure):
 installed gir1.2-dee-1.0 package post-installation script subprocess returned e                                   rror exit status 127
Setting up python3-minimal (3.8.2-0ubuntu2) ...
Traceback (most recent call last):
  File "/usr/bin/py3compile", line 28, in <module>
    import struct
  File "/usr/lib/python3.8/struct.py", line 13, in <module>
    from _struct import *
ModuleNotFoundError: No module named '_struct'
dpkg: error processing package python3-minimal (--configure):
 installed python3-minimal package post-installation script subprocess returned                                    error exit status 1
No apport report written because MaxReports is reached already
                                                              Errors were encoun                                   tered while processing:
 pychess
 python-pkg-resources
 gir1.2-dee-1.0
 python3-minimal
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

Reinstalling with sudo apt install --reinstall -f python3-all does not help either.

  $ sudo apt install --reinstall -f python3-all
  Reading package lists... Done
  Building dependency tree
  Reading state information... Done
  0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
  4 not fully installed or removed.
  Need to get 0 B/131 kB of archives.
  After this operation, 0 B of additional disk space will be used.
  (Reading database ... 1052975 files and directories currently installed.)
  Preparing to unpack .../python3-all_3.8.2-0ubuntu2_arm64.deb ...
  Unpacking python3-all (3.8.2-0ubuntu2) over (3.8.2-0ubuntu2) ...
  Setting up pychess (0.12.2-1build1) ...
  /var/lib/dpkg/info/pychess.postinst: 6: pycompile: not found
  dpkg: error processing package pychess (--configure):
   installed pychess package post-installation script subprocess returned error exit status 127
  dpkg: error processing package python-pkg-resources (--configure):
   package is in a very bad inconsistent state; you should
   reinstall it before attempting configuration
  Setting up python3-all (3.8.2-0ubuntu2) ...
  Setting up gir1.2-dee-1.0 (1.2.7+17.10.20170616-4ubuntu6) ...
  /var/lib/dpkg/info/gir1.2-dee-1.0.postinst: 6: pycompile: not found
  dpkg: error processing package gir1.2-dee-1.0 (--configure):
   installed gir1.2-dee-1.0 package post-installation script subprocess returned error exit status 127
  Setting up python3-minimal (3.8.2-0ubuntu2) ...
  Traceback (most recent call last):
    File "/usr/bin/py3compile", line 28, in <module>
      import struct
    File "/usr/lib/python3.8/struct.py", line 13, in <module>
      from _struct import *
  ModuleNotFoundError: No module named '_struct'
  dpkg: error processing package python3-minimal (--configure):
   installed python3-minimal package post-installation script subprocess returned error exit status 1
  No apport report written because MaxReports is reached already
                                                                Errors were encountered while processing:
   pychess
   python-pkg-resources
   gir1.2-dee-1.0
   python3-minimal
  needrestart is being skipped since dpkg has failed
  E: Sub-process /usr/bin/dpkg returned an error code (1)

sudo dpkg --configure -a and sudo apt -f install, result in the same errors.
The dpkg recovery instructions (as for i.e. How to reinstall dpkg) don't work either since there aren't i.e. the required files at /var/cache/apt/archives/.

I tried to get the .deb packages straight from Ubuntu's website, extract them and install them separately.

Given the fact that according to the forum posts that I've been going through, people have been breaking their Ubuntu and other Debian-based systems for at least the past ten years whenever something has gone haywire with Python - be it user error or a package breaking things, would it be a bit safer to somehow make a fundamental fix in the way the dpkg/apt architecture is handled in Linux and isolate whatever Python modules and dependencies they need to run from the rest of the operating system? Or to create some sort of a fail-safe system restore method in cases of these kinds of grievous situations which seem to be quite prevalent.

score 5 · Answer 1 · answered Sep 07 '20 at 14:01

I responded to your previous post but it appears it being locked caused some issues.

As I mentioned in my original post, you can manually extract the relevant .deb files and rebuild a cobbled together Python to bootstrap yourself. It does appear that https://packages.ubuntu.com/focal/python3 does not have any mirrors listed for arm64 but, after some brief searching, I found https://ubuntu.pkgs.org/20.04/ubuntu-main-arm64/python3_3.8.2-0ubuntu2_arm64.deb.html which gives you all the package names and then points you to http://ports.ubuntu.com/pool/main/p/python3.8/ for download (this mirror has all the arm64 packages). You'll have to start at the "python3" package and manually drill down all the dependencies to grab them all. It looks like py3compile and py3clean are both in python3-minimal package

Secondly, and I say this having made many mistakes myself, there are innumerable things people can do with sudo that can render a system inoperable and there are no realistic safeguards. Should sudo rm require a warning/confirmation every time? How about sudo chmod? For every post about someone breaking their system with Python problems, I can probably point to as many where people did something bad with permissions or deleted the wrong file.

score 3 · Answer 2 · answered Aug 27 '20 at 14:42

I will quote from experience - if you attempt to upgrade Python from source in your system, you will torch your system in a unfixable way. The fastest and most easiest solution in these cases is to reinstall your system or restore from a known good backup image and DO NOT attempt to upgrade Python from source.

The "reinstall" answer applies to all OS versions, not just ISO installs.

Restoring system python from your 'backup image' is going to be pain, so you may as well just do a full system restore from your backup. Or reinstall from scratch.

As for getting Newer python for your own needs, you may want to look into using PyEnv which is a project/tool that lets you have multiple Python versions installed in User Space without affecting your system Python and thereby doesn't break your system python. Because updating System level python will leave you with a broken system in many cases.

score 0 · Answer 3 · answered Sep 08 '20 at 17:59

Yes, thanks to @Brian Turek , my system's up and running again! I got it working by SSH'ing on to my box and grabbing the necessary .deb files from the link he pointed out ( http://ports.ubuntu.com/pool/main/p/python3.8/ ) with wget.

First had to install (sudo dpkg -i) the libpython3.8-stdlib_3.8.5-2_arm64.deb package, then the python3.8-minimal_3.8.5-2_arm64.deb package, and only THEN the python3.8_3.8.5-2_arm64.deb package would install because of the dependencies being off balance.

NOTE to future generations (if there are any): The installation procedure for a system with a broken Python setup may vary; just be careful to look keep a close lookout for the error messages that occur during the dpkginstallation with each package -- if it won't install because it's dependent on another version of some other dependency, look that up from the repository URL that Brian originally linked to, install the required package first, if that gives you more nags, then you gotta dig in deeper and install those broken dependencies first to "unclog the toilet bowl from the Python that's in there", rinse and repeat until all's sorted out.

Nonetheless, I finally made it, my system is back up and running and I'm happily installing all the packages and unattended updates that had been hanging on due to this issue. Thank you Brian Turek for your help! <3 You saved me many years of psychotherapy.

P.S. All you haters out there who think the only way to fix a system with a broken Python is to just wipe it all away and start from scratch, "Touché!" :-D

Python 3.8.x problems in 20.04

3 Answers3