2

Overview / Scenario

I already have a fully working autoinstall configuration that is using the default apt configuration (no apt config in cloud-init user data). The source image is using the Ubuntu 20.04 cloud image.

My goal is to edit the autoinstall config to use private Apt repos (the repos already exist). I must ensure the installer never reaches out to Ubuntu servers for Apt packages and only hits custom Apt repos. The repos are already established and have been working for a long time, I just need to get the installer to point to them.

I have already reviewed How to add apt repository with Ubuntu autoinstall

Here is what I've added to the autoinstall cloud init:

Note: the below mentioned /cdrom/apt-key.gpg file is present and valid for the repo (tested thoroughly).

Note: The repos are over https using private CA. Since I can't install the ca-certificates package yet during install before configuring Apt, I'm just forcing it to be trusted. trusted=yes didn't work, I had to also add the apt.conf.d/99verify-peer.conf as shown below to get it to work. This may be a cause for the issue but it works fine manually at an install shell. I have the certs and can inject them into the install media, so if there is a better way to get the installer instance to trust my CA cert without having to force trust, I'm open to that change!

  early-commands:
    - 'mkdir -p /etc/apt'
    - 'cp /cdrom/apt-key.gpg /etc/apt/fresh-ubuntu-key.gpg'
    - 'touch /etc/apt/apt.conf.d/99verify-peer.conf'
    - 'echo >>/etc/apt/apt.conf.d/99verify-peer.conf "Acquire { https::Verify-Peer false }"'
  apt:
    preserve_sources_list: false
    sources_list: |
      deb [trusted=yes signed-by=/etc/apt/fresh-ubuntu-key.gpg] https://repo.internal.example.com/aptly/focal focal-ver221 main
      deb [trusted=yes signed-by=/etc/apt/fresh-ubuntu-key.gpg] https://repo.internal.example.com/aptly/focal focal-security-ver221 main
      deb [trusted=yes signed-by=/etc/apt/fresh-ubuntu-key.gpg] https://repo.internal.example.com/aptly/focal focal-updates-ver221 main
      deb [trusted=yes signed-by=/etc/apt/fresh-ubuntu-key.gpg] https://repo.internal.example.com/aptly/focal focal-backports-ver221 main

Problem / Troubleshooting

When trying to install, it fails with the following:

finish: subiquity/Updates/apply_autoinstall_config
start: subiquity/Late/apply_autoinstall_config
finish: subiquity/Late/apply_autoinstall_config
start: subiquity/Shutdown/apply_autoinstall_config
finish: subiquity/Shutdown/apply_autoinstall_config
finish: subiquity/apply_autoinstall_config
start:   subiquity/Install/install/configure_apt: configuring apt
start: subiquity/Meta/status_GET
start: subiquity/Meta/status_GET
start: subiquity/Meta/status_GET
start: subiquity/Meta/status_GET
start: subiquity/Mirror/cmd-apt-config: curtin command apt-config
finish: subiquity/Mirror/cmd-apt-config: curtin command apt-config
start:     subiquity/Install/install/configure_apt/cmd-in-target: curtin command in-target
finish:   subiquity/Install/install/configure_apt: Command '['/snap/subiquity/3119/usr/bin/python3.8', '-m', 'curtin', '--showtrace', '-vvv', '--set', 'json:reporting={"subiquity": {"type": "journald", "identifier": "curtin_event.2752.2"}}', 'in-target', '-t', '/tmp/tmpzyqgk825/mount', '--', 'apt-get', 'update']' returned non-zero exit status 100.
finish: subiquity/Install/install: Command '['/snap/subiquity/3119/usr/bin/python3.8', '-m', 'curtin', '--showtrace', '-vvv', '--set', 'json:reporting={"subiquity": {"type": "journald", "identifier": "curtin_event.2752.2"}}', 'in-target', '-t', '/tmp/tmpzyqgk825/mount', '--', 'apt-get', 'update']' returned non-zero exit status 100.
start:   subiquity/ErrorReporter/1669932701.344982386.install_fail/add_info
finish:     subiquity/Install/install/configure_apt/cmd-in-target: curtin command in-target
finish:   subiquity/ErrorReporter/1669932701.344982386.install_fail/add_info: written to /var/crash/1669932701.344982386.install_fail.crash
An error occurred. Press enter to start a shell

I tried finding anything meaningful in /var/crash/1669932701.344982386.install_fail.crash but it's a massive file and hard to parse. Doesn't seem to have any additional failure info that I can see.

Important Note: After it fails, I can drop into a shell, update /etc/apt/sources.list manually and apt update works perfectly fine. So I know that the installer instance CAN properly communicate with my private repos. So it seems the issue has to be something specific during autoinstall/cloud-init.

Please help if you have any ideas, thanks!!

Rino Bino
  • 165

1 Answers1

1

tl;dr

You could try something like

#cloud-config
autoinstall:
  apt:
    preserve_sources_list: false
    sources_list: |
      deb [trusted=yes] https://repo.internal.example.com/aptly/focal focal-ver221 main
      deb [trusted=yes] https://repo.internal.example.com/aptly/focal focal-security-ver221 main
      deb [trusted=yes] https://repo.internal.example.com/aptly/focal focal-updates-ver221 main
      deb [trusted=yes] https://repo.internal.example.com/aptly/focal focal-backports-ver221 main
     conf: |
      Acquire::https::repo.internal.example.com::Verify-Peer "false";
      Acquire::https::repo.internal.example.com::Verify-Host "false";

reasoning

To start, the early-commands in your autoinstall config are not doing what you seem to think. The commands are configuring apt within the installer environment. However, the apt commands during installation generally happen in a different chroot environment. The logs you shared show this. The failing command is partially in-target -t /tmp/tmpzyqgk825/mount apt-get update.

The second issue is that the repository has a custom gpg key. The only way to provide a custom key that I am aware of is using the syntax in the other answer you referenced. You can certainly also add the sources.list option trusted=yes to avoid a failure from the missing gpg key, but that is probably not a good configuration in the long run.

The third issue is that a private CA signed the https certificate. There is no provided way to trust this private CA. It looks like a bug has been filed requesting this. It also looks like there is an apt configuration option that can be used to ignore the certificate problems. Again, that is probably not a good configuration in the long run.

It is probably easiest to late-commands to install the custom gpg key and the private CA, and reconfigure apt.

update

As noted in the comments, this solution did not work. I'm sharing what I found here. I tested the install process using Ubuntu 22.04 (subiquity 22.04.2).

  • subiquity creates /var/log/installer/subiquity-curtin-apt.conf. It basically contains the autoinstall apt section (and potentially proxy information).
  • subiquity runs curtin to configure apt using /var/log/installer/subiquity-curtin-apt.conf in the /tmp/tmpXXX overlay file system. If my autoinstall snippet above is used then curtin will create /tmp/tmpXXX/etc/apt/apt.conf.d/94curtin-config.
  • subiquity creates /var/log/installer/subiquity-curtin-install.conf. This does not contain an apt section. Presumably, because apt is configured already.
  • subiquity runs curtin to perform the install using /var/log/installer/subiquity-curtin-install.conf. This will copy /tmp/tmpXXX to /target. This creates /target/etc/apt/apt.conf.d/94curtin-config.
  • At a later stage during the install, curtin configures apt. Because the install configuration does not contain the apt section curtin will actually delete 94curtin-config.
  • Subsequent apt commands that require the apt configuration in 94curtin-config will fail and cause the installation to fail.
  • If the installation does not fail because of the missing 94curtin-config file then a later step in subiquity will copy /tmp/tmpXXX/etc/apt to /target/etc/apt and the resulting installation will contain 94curtin-config.

In short, do not depend on the apt conf settings using autoinstall.

Andrew Lowther
  • 7,251