Upgrading AWS Ubuntu Server

MikeJones · Oct 27, 2020

I am running a Ubuntu 16.04.7 LTS EC2 Instance on AWS and I am having issues upgrading it.

I have tried:

`sudo apt update

sudo apt upgrade

sudo reboot`

And there are no issues until I try:

`sudo do-release-upgrade`

And I get this error:

Code:

<s>```</s><i>
</i>Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/bionic", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeMain.py", line 238, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 2072, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1981, in fullUpgrade
    if not self.askDistUpgrade():
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1153, in askDistUpgrade
    changes = self.calcDistUpgrade()
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1120, in calcDistUpgrade
    if not self.cache.installTasks(self.tasks):
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeCache.py", line 856, in installTasks
    pkg.mark_install()
  File "/usr/lib/python3/dist-packages/apt/package.py", line 1356, in mark_install
    fixer.resolve(True)
SystemError: E:Unable to correct problems, you have held broken packages.
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/problem_report.py", line 497, in add_to_existing
    self.write(f)
  File "/usr/lib/python3/dist-packages/problem_report.py", line 450, in write
    block = f.read(1048576)
  File "/usr/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Original exception was:
Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/bionic", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeMain.py", line 238, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 2072, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1981, in fullUpgrade
    if not self.askDistUpgrade():
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1153, in askDistUpgrade
    changes = self.calcDistUpgrade()
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeController.py", line 1120, in calcDistUpgrade
    if not self.cache.installTasks(self.tasks):
  File "/tmp/ubuntu-release-upgrader-_29ga8ky/DistUpgrade/DistUpgradeCache.py", line 856, in installTasks
    pkg.mark_install()
  File "/usr/lib/python3/dist-packages/apt/package.py", line 1356, in mark_install
    fixer.resolve(True)
SystemError: E:Unable to correct problems, you have held broken packages.
=== Command terminated with exit status 1 (Thu Oct 22 03:03:11 2020) ===<i>
</i><e>```</e>

Abhi · Oct 27, 2020

It looks like a Dependency issue. If you have proper backups, I would suggest

`sudo apt autoremove`

Have you seen this ?

https://askubuntu.com/questions/1214292/upgrade-path-from-16-04-to-20-04-lts-without-gui-change

Most probably you will have to do it the old fashioned way. ie, upgrade to 18.04 and then to 20.04

https://askubuntu.com/questions/126...-ubuntu-16-04-lts-to-20-04-in-virtual-machine

Abhi · Oct 27, 2020

Official guides from Ubuntu

https://askubuntu.com/questions/126...-ubuntu-16-04-lts-to-20-04-in-virtual-machine

16.04 to 18.04 by DO, hopefully it will work on AWS too

https://www.digitalocean.com/community/tutorials/how-to-upgrade-to-ubuntu-18-04

MikeJones · Oct 27, 2020

@Abhi awesome thanks I will try the digital ocean guide too. They have great tutorials.

I will let you know how it goes!

MikeJones · Oct 28, 2020

So I followed the Digital Ocean steps and after running this:

`sudo do-release-upgrade`

It failed again with:

`Broken landscape-common:amd64 Depends on python3-twisted [ amd64 ] < none -> 17.9.0-2ubuntu0.1 > ( python )

Considering python3-twisted:amd64 1 as a solution to landscape-common:amd64 9999

Reinst Failed because of python3-zope.interface:amd64

MarkKeep python3-twisted [ amd64 ] < none -> 17.9.0-2ubuntu0.1 > ( python ) FU=0

Considering python3-twisted:amd64 1 as a solution to landscape-common:amd64 9999

Done

Traceback (most recent call last):

File "/tmp/ubuntu-release-upgrader-a7x268x1/bionic", line 8, in

sys.exit(main())

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeMain.py", line 238, in main

if app.run():

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 2072, in run

return self.fullUpgrade()

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1981, in fullUpgrade

if not self.askDistUpgrade():

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1153, in askDistUpgrade

changes = self.calcDistUpgrade()

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1120, in calcDistUpgrade

if not self.cache.installTasks(self.tasks):

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeCache.py", line 856, in installTasks

pkg.mark_install()

File "/usr/lib/python3/dist-packages/apt/package.py", line 1356, in mark_install

fixer.resolve(True)

SystemError: E:Unable to correct problems, you have held broken packages.

Error in sys.excepthook:

Traceback (most recent call last):

File "/usr/lib/python3/dist-packages/problem_report.py", line 497, in add_to_existing

self.write(f)

File "/usr/lib/python3/dist-packages/problem_report.py", line 450, in write

block = f.read(1048576)

File "/usr/lib/python3.5/codecs.py", line 321, in decode

(result, consumed) = self._buffer_decode(data, self.errors, final)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Original exception was:

Traceback (most recent call last):

File "/tmp/ubuntu-release-upgrader-a7x268x1/bionic", line 8, in

sys.exit(main())

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeMain.py", line 238, in main

if app.run():

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 2072, in run

return self.fullUpgrade()

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1981, in fullUpgrade

if not self.askDistUpgrade():

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1153, in askDistUpgrade

changes = self.calcDistUpgrade()

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeController.py", line 1120, in calcDistUpgrade

if not self.cache.installTasks(self.tasks):

File "/tmp/ubuntu-release-upgrader-a7x268x1/DistUpgrade/DistUpgradeCache.py", line 856, in installTasks

pkg.mark_install()

File "/usr/lib/python3/dist-packages/apt/package.py", line 1356, in mark_install

fixer.resolve(True)

SystemError: E:Unable to correct problems, you have held broken packages.

=== Command detached from window (Wed Oct 28 00:45:53 2020) ===

=== Command terminated with exit status 1 (Wed Oct 28 00:46:03 2020) ===`

Abhi · Oct 28, 2020

Were you able to move smoothly from 16.04 to 18.04 ? I think the following link and answer by _user535733_ might be helpful. The person replied is a part of the Ubuntu team ( I think so )

https://askubuntu.com/questions/126...-ubuntu-16-04-lts-to-20-04-in-virtual-machine

MikeJones · Oct 28, 2020

@Abhi nope, still stuck.

I asked for help on the [official Ubuntu Forum](https://ubuntuforums.org/showthread.php?t=2452758&p=13995995#post13995995) and someone asked me to run this command:

`apt-cache policy landscape-common:amd64
`

Which I did and gave them the output, but not sure what next step is yet.

Abhi · Oct 29, 2020

From what I can see you will have to upgrade to 18.04 first and then upgrade to 20.04. I assume that is what you have done so far. If no, kindly do update.

It would be easier if you can fresh install 20.04 and avoid all these troubles. That way you have an error-free install.

MikeJones · Oct 29, 2020

@Abhi I have not upgraded to 18.04 yet. I think thats where my issue lies.

> @Abhi It would be easier if you can fresh install 20.04 and avoid all these troubles. That way you have an error-free install.

If I do that I have to reconfigure my entire nginx settings and somehow import all my flarum data into the new server.... I think that seems just as much of a hassle, no?

Abhi · Oct 29, 2020

I understand your trouble. From the error data you posted, the issues seems to be broken packages / unmet dependencies. The issue is at the point from where you are trying to updating. ie, something is already broken on your server and from there you are trying an upgrade. Is there any chance you have a snapshot a bit older than the point from which you are trying ? Yes, it would loose you a few forum posts. But maybe a day or two old, will is cost you too much. I would suggest give it a try. I really hoped that one of those guides might have pulled you out.

PS : Did you try sudo apt autoremove ( backup please ) ? What was the result ?

Abhi · Oct 29, 2020

There might be hope

https://www.rosehosting.com/blog/how-to-fix-broken-packages-on-ubuntu-16-04-and-debian-9/

http://www.iasptk.com/ubuntu-fix-broken-package-best-solution/

I have not tried these, if interested, take a look

MikeJones · Nov 19, 2020

So I think my issue with upgrading might be this:

`Reinst Failed because of python3-zope.interface:amd64 MarkKeep python3-twisted [ amd64 ]`

Any more thoughts on this?

Abhi · Nov 19, 2020

Could be because of partially upgrade packages.

I haven't tried :

1.

`sudo apt-get install --fix-broken`

or

2.

remove all Python and install it afresh

`sudo apt-get purge python* && sudo apt-get autoclean && sudo apt-get install python*`

phenomlab · Nov 19, 2020

@MikeJones Mike, this seems like the artefact of a removed PPA relating to Certbot. Have you removed any repositories recently before attempting the upgrade ?

MikeJones · Nov 20, 2020

@phenomlab I dont believe so but is there anyway to check/fix?

phenomlab · Nov 20, 2020

@MikeJones What is the result of ```sudo apt install certbot python3-certbot-nginx```

MikeJones · Nov 20, 2020

@phenomlab

I got this:

Code:

<s>```</s><i>
</i>certbot is already the newest version (0.31.0-1+ubuntu16.04.1+certbot+1).
python3-certbot-nginx is already the newest version (0.31.0-1+ubuntu16.04.1+cert               bot+1).
python3-certbot-nginx set to manually installed.
The following packages were automatically installed and are no longer required:
  javascript-common jsonlint libargon2-0 libjs-excanvas libvpx3 libzip5
  linux-aws-headers-4.4.0-1054 linux-aws-headers-4.4.0-1055
  linux-aws-headers-4.4.0-1057 linux-aws-headers-4.4.0-1060
  linux-aws-headers-4.4.0-1061 linux-aws-headers-4.4.0-1062
  linux-aws-headers-4.4.0-1065 linux-aws-headers-4.4.0-1066
  linux-aws-headers-4.4.0-1069 linux-aws-headers-4.4.0-1070
  linux-aws-headers-4.4.0-1072 linux-aws-headers-4.4.0-1074
  linux-aws-headers-4.4.0-1075 linux-aws-headers-4.4.0-1077
  linux-aws-headers-4.4.0-1079 linux-aws-headers-4.4.0-1083
  linux-aws-headers-4.4.0-1084 linux-aws-headers-4.4.0-1085
  linux-aws-headers-4.4.0-1087 linux-aws-headers-4.4.0-1088
  linux-aws-headers-4.4.0-1090 linux-aws-headers-4.4.0-1092
  linux-aws-headers-4.4.0-1094 mercurial mercurial-common php-cli-prompt
  php-composer-semver php-composer-spdx-licenses php-json-schema
  php-symfony-console php-symfony-filesystem php-symfony-finder
  php-symfony-process
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 40 not upgraded.
[email protected]:~$<i>
</i><e>```</e>

MikeJones · Nov 24, 2020

So good news, I figured it out! I had to uninstall python3 and python-certbot-nginx and then I was able to upgrade!

Unfortunately now that that problem is solved, I am onto the next one! NGINX doesnt seem to be running:https://askubuntu.com/questions/129...ntu-version-16-to-20-and-nginx-is-not-working

phenomlab · Nov 24, 2020

> @MikeJones So good news, I figured it out! I had to uninstall python3 and python-certbot-nginx and then I was able to upgrade!

That was going to be my next suggestion

You beat me to it - that's what happens when you "snooze" (I wish) I suppose

In relation to NGINX not starting, what's the output of `journalctl -xe` and anything in `journalctl | tail` ?

This is a bit old, but may still be useful

https://www.digitalocean.com/community/questions/can-t-start-nginx-job-for-nginx-service-failed

MikeJones · Nov 24, 2020

@phenomlab

For journalctl -xe:

`-- A start job for unit session-70.scope has finished successfully.
--
-- The job identifier is 3818.

Nov 24 23:50:02 ip-172-31-66-31 systemd[7236]: Reached target Main User Target.

-- Subject: A start job for unit UNIT has finished successfully

-- Defined-By: systemd
-- Support:http://www.ubuntu.com/support
--
-- A start job for unit UNIT has finished successfully.
--
-- The job identifier is 1.

Nov 24 23:50:02 ip-172-31-66-31 systemd[7236]: Startup finished in 52ms.

-- Subject: User manager start-up is now complete

-- Defined-By: systemd
-- Support:http://www.ubuntu.com/support
--
-- The user manager instance for user 1000 has been started. All services queued

-- for starting have been started. Note that other services might still be starting
-- up or be started at any later time.
--
-- Startup of the manager took 52538 microseconds.

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 16: Deprecated option UsePrivilegeSeparation

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 19: Deprecated option KeyRegenerationInterval

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 20: Deprecated option ServerKeyBits

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 31: Deprecated option RSAAuthentication

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 38: Deprecated option RhostsRSAAuthentication

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: reprocess config line 31: Deprecated option RSAAuthenti>

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: reprocess config line 38: Deprecated option RhostsRSAAu>

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Invalid user jenkins from 104.248.243.134 port 41200

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Received disconnect from 104.248.243.134 port 41200:11:>

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Disconnected from invalid user jenkins 104.248.243.134 >

lines 2040-2070/2070 (END)

`

and for journalctl | tail:

`[email protected]:~$ journalctl | tail

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 16: Deprecated option UsePrivilegeSeparation

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 19: Deprecated option KeyRegenerationInterval

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 20: Deprecated option ServerKeyBits

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 31: Deprecated option RSAAuthentication

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: rexec line 38: Deprecated option RhostsRSAAuthentication

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: reprocess config line 31: Deprecated option RSAAuthentication

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: reprocess config line 38: Deprecated option RhostsRSAAuthentication

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Invalid user jenkins from 104.248.243.134 port 41200

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Received disconnect from 104.248.243.134 port 41200:11: Normal Shutdown, Thank you for playing [preauth]

Nov 24 23:50:21 ip-172-31-66-31 sshd[7278]: Disconnected from invalid user jenkins 104.248.243.134 port 41200 [preauth]

[email protected]:~$
`

Upgrading AWS Ubuntu Server

Webmaster Forums, Forum Promotion, SEO, Domain Names, Hosting & More!

AdminBrave is a community with useful content and resources. Connect with fellow webmasters. Grow your network. Stay informed. Learn and share with us!

MikeJones

Abhi

Abhi

MikeJones

MikeJones

Abhi

MikeJones

Abhi

MikeJones

Abhi

Abhi

MikeJones

Abhi

phenomlab

MikeJones

phenomlab

MikeJones

MikeJones

phenomlab

MikeJones

Similar threads

Forum statistics

Newest members