Revisions to How to upgrade all Python packages with pip

fixed references in "Detals"

Source Link

edit approved Feb 1, 2021 at 13:21

797
5
20

More Robust Solution

For pip3, use this:

pip3 freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip3 install -U \1/p' |sh

For pip, just remove the 3s as such:

pip freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip install -U \1/p' |sh

OS X Oddity

OS X, as of July 2017, ships with a very old version of sed (a dozen years old). To get extended regular expressions, use -E instead of -r in the solution above.

Solving Issues with Popular Solutions

This solution is well designed and tested¹, whereas there are problems with even the most popular solutions.

Portability issues due to changing pip command line features
Crashing of xargs because of common pip or pip3 child process failures
Crowded logging from the raw xargs output
Relying on a Python-to-OS bridge while potentially upgrading it³

The above command uses the simplest and most portable pip syntax in combination with sed and sh to overcome these issues completely. Details of the sed operation can be scrutinized with the commented version².

Details

2 [1] Tested and regularly used in a Linux 4.8.16-200.fc24.x86_64 cluster and tested on five other Linux/Unix flavors. It also runs on Cygwin64 installed on Windows 10. Testing on iOS is needed.

2 [2] To see the anatomy of the command more clearly, this is the exact equivalent of the above pip3 command with comments:

# Match lines from pip's local package list output
# that meet the following three criteria and pass the
# package name to the replacement string in group 1.
# (a) Do not start with invalid characters
# (b) Follow the rule of no white space in the package names
# (c) Immediately follow the package name with an equal sign
sed="s/^([^=# \t\\][^ \t=]*)=.*"

# Separate the output of package upgrades with a blank line
sed="$sed/echo"

# Indicate what package is being processed
sed="$sed; echo Processing \1 ..."

# Perform the upgrade using just the valid package name
sed="$sed; pip3 install -U \1"

# Output the commands
sed="$sed/p"

# Stream edit the list as above
# and pass the commands to a shell
pip3 freeze --local | sed -rn "$sed" | sh

[3] Upgrading a Python or PIP component that is also used in the upgrading of a Python or PIP component can be a potential cause of a deadlock or package database corruption.

More Robust Solution

For pip3, use this:

pip3 freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip3 install -U \1/p' |sh

For pip, just remove the 3s as such:

pip freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip install -U \1/p' |sh

OS X Oddity

OS X, as of July 2017, ships with a very old version of sed (a dozen years old). To get extended regular expressions, use -E instead of -r in the solution above.

Solving Issues with Popular Solutions

This solution is well designed and tested¹, whereas there are problems with even the most popular solutions.

Portability issues due to changing pip command line features
Crashing of xargs because of common pip or pip3 child process failures
Crowded logging from the raw xargs output
Relying on a Python-to-OS bridge while potentially upgrading it³

The above command uses the simplest and most portable pip syntax in combination with sed and sh to overcome these issues completely. Details of the sed operation can be scrutinized with the commented version².

Details

2 Tested and regularly used in a Linux 4.8.16-200.fc24.x86_64 cluster and tested on five other Linux/Unix flavors. It also runs on Cygwin64 installed on Windows 10. Testing on iOS is needed.

2 To see the anatomy of the command more clearly, this is the exact equivalent of the above pip3 command with comments:

# Match lines from pip's local package list output
# that meet the following three criteria and pass the
# package name to the replacement string in group 1.
# (a) Do not start with invalid characters
# (b) Follow the rule of no white space in the package names
# (c) Immediately follow the package name with an equal sign
sed="s/^([^=# \t\\][^ \t=]*)=.*"

# Separate the output of package upgrades with a blank line
sed="$sed/echo"

# Indicate what package is being processed
sed="$sed; echo Processing \1 ..."

# Perform the upgrade using just the valid package name
sed="$sed; pip3 install -U \1"

# Output the commands
sed="$sed/p"

# Stream edit the list as above
# and pass the commands to a shell
pip3 freeze --local | sed -rn "$sed" | sh

[3] Upgrading a Python or PIP component that is also used in the upgrading of a Python or PIP component can be a potential cause of a deadlock or package database corruption.

More Robust Solution

For pip3, use this:

pip3 freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip3 install -U \1/p' |sh

For pip, just remove the 3s as such:

pip freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip install -U \1/p' |sh

OS X Oddity

OS X, as of July 2017, ships with a very old version of sed (a dozen years old). To get extended regular expressions, use -E instead of -r in the solution above.

Solving Issues with Popular Solutions

This solution is well designed and tested¹, whereas there are problems with even the most popular solutions.

Portability issues due to changing pip command line features
Crashing of xargs because of common pip or pip3 child process failures
Crowded logging from the raw xargs output
Relying on a Python-to-OS bridge while potentially upgrading it³

The above command uses the simplest and most portable pip syntax in combination with sed and sh to overcome these issues completely. Details of the sed operation can be scrutinized with the commented version².

Details

[1] Tested and regularly used in a Linux 4.8.16-200.fc24.x86_64 cluster and tested on five other Linux/Unix flavors. It also runs on Cygwin64 installed on Windows 10. Testing on iOS is needed.

[2] To see the anatomy of the command more clearly, this is the exact equivalent of the above pip3 command with comments:

# Match lines from pip's local package list output
# that meet the following three criteria and pass the
# package name to the replacement string in group 1.
# (a) Do not start with invalid characters
# (b) Follow the rule of no white space in the package names
# (c) Immediately follow the package name with an equal sign
sed="s/^([^=# \t\\][^ \t=]*)=.*"

# Separate the output of package upgrades with a blank line
sed="$sed/echo"

# Indicate what package is being processed
sed="$sed; echo Processing \1 ..."

# Perform the upgrade using just the valid package name
sed="$sed; pip3 install -U \1"

# Output the commands
sed="$sed/p"

# Stream edit the list as above
# and pass the commands to a shell
pip3 freeze --local | sed -rn "$sed" | sh

[3] Upgrading a Python or PIP component that is also used in the upgrading of a Python or PIP component can be a potential cause of a deadlock or package database corruption.

Active reading [<https://en.wikipedia.org/wiki/Mac_OS_X>]. Made compliant with the Jon Skeet Decree - <https://twitter.com/PeterMortensen/status/976400000942034944>.

Source Link

edited Oct 6, 2020 at 22:10

Peter Mortensen

31.4k
22
110
134

For pip3pip3, use this:

For pippip, just remove the 3s as such:

OSXOS X Oddity

OSXOS X, as of July 2017, ships with a very old version of sedsed (a dozen years old). ToTo get extended regular expressions, use -E-E instead of -r-r in the solution above.

Portability issues due to changing pip command line features
Crashing of xargsxargs because of common pip or pip3 child process failures
Crowded logging from the raw xargsxargs output
Relying on a Python-to-OS bridge while potentially upgrading it³

The above command uses the simplest and most portable pippip syntax in combination with sedsed and shsh to overcome these issues completely. DetailsDetails of sedthe sed operation can be scrutinized with the commented version².

[1]2 Tested and regularly used in a Linux 4.8.16-200.fc24.x86_64 cluster and tested on five other Linux/Unix flavors. It also runs on Cygwin64 installed on Windows 10. Testing on iOS is needed.

[2]2 To see the anatomy of the command more clearly, this is the exact equivalent of the above pip3 command with comments:

# matchMatch lines from pip's local package list output
# that meet the following three criteria and pass the
# package name to the replacement string in group 1.
# (a) Do not start with invalid characters
# (b) Follow the rule of no white space in the package names
# (c) Immediately follow the package name with an equal sign
sed="s/^([^=# \t\\][^ \t=]*)=.*"

# separateSeparate the output of package upgrades with a blank line
sed="$sed/echo"

# indicateIndicate what package is being processed
sed="$sed; echo Processing \1 ..."

# performPerform the upgrade using just the valid package name
sed="$sed; pip3 install -U \1"

# outputOutput the commands
sed="$sed/p"

# streamStream edit the list as above
# and pass the commands to a shell
pip3 freeze --local |sed| sed -rn "$sed" |sh| sh

For pip3 use this:

For pip, just remove the 3s as such:

OSX Oddity

OSX, as of July 2017, ships with a very old version of sed (a dozen years old). To get extended regular expressions, use -E instead of -r in the solution above.

Portability issues due to changing pip command line features
Crashing of xargs because common pip or pip3 child process failures
Crowded logging from the raw xargs output
Relying on a Python-to-OS bridge while potentially upgrading it³

The above command uses the simplest and most portable pip syntax in combination with sed and sh to overcome these issues completely. Details of sed operation can be scrutinized with the commented version².

[1] Tested and regularly used in a Linux 4.8.16-200.fc24.x86_64 cluster and tested on five other Linux/Unix flavors. It also runs on Cygwin64 installed on Windows 10. Testing on iOS is needed.

[2] To see the anatomy of the command more clearly, this is the exact equivalent of the above pip3 command with comments:

# match lines from pip's local package list output
# that meet the following three criteria and pass the
# package name to the replacement string in group 1.
# (a) Do not start with invalid characters
# (b) Follow the rule of no white space in the package names
# (c) Immediately follow the package name with an equal sign
sed="s/^([^=# \t\\][^ \t=]*)=.*"

# separate the output of package upgrades with a blank line
sed="$sed/echo"

# indicate what package is being processed
sed="$sed; echo Processing \1 ..."

# perform the upgrade using just the valid package name
sed="$sed; pip3 install -U \1"

# output the commands
sed="$sed/p"

# stream edit the list as above
# and pass the commands to a shell
pip3 freeze --local |sed -rn "$sed" |sh

For pip3, use this:

For pip, just remove the 3s as such:

OS X Oddity

OS X, as of July 2017, ships with a very old version of sed (a dozen years old). To get extended regular expressions, use -E instead of -r in the solution above.

Portability issues due to changing pip command line features
Crashing of xargs because of common pip or pip3 child process failures
Crowded logging from the raw xargs output
Relying on a Python-to-OS bridge while potentially upgrading it³

The above command uses the simplest and most portable pip syntax in combination with sed and sh to overcome these issues completely. Details of the sed operation can be scrutinized with the commented version².

2 Tested and regularly used in a Linux 4.8.16-200.fc24.x86_64 cluster and tested on five other Linux/Unix flavors. It also runs on Cygwin64 installed on Windows 10. Testing on iOS is needed.

2 To see the anatomy of the command more clearly, this is the exact equivalent of the above pip3 command with comments:

# Match lines from pip's local package list output
# that meet the following three criteria and pass the
# package name to the replacement string in group 1.
# (a) Do not start with invalid characters
# (b) Follow the rule of no white space in the package names
# (c) Immediately follow the package name with an equal sign
sed="s/^([^=# \t\\][^ \t=]*)=.*"

# Separate the output of package upgrades with a blank line
sed="$sed/echo"

# Indicate what package is being processed
sed="$sed; echo Processing \1 ..."

# Perform the upgrade using just the valid package name
sed="$sed; pip3 install -U \1"

# Output the commands
sed="$sed/p"

# Stream edit the list as above
# and pass the commands to a shell
pip3 freeze --local | sed -rn "$sed" | sh

Added section to allow the handling of very old OSX version of sed.

Source Link

edited Jul 30, 2017 at 0:46

Douglas Daseeco

3.7k
25
29

More Robust Solution

For pip3 use this:

pip3 freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip3 install -U \1/p' |sh

For pip, just remove the 3s as such:

pip freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip install -U \1/p' |sh

OSX Oddity

OSX, as of July 2017, ships with a very old version of sed (a dozen years old). To get extended regular expressions, use -E instead of -r in the solution above.

Solving Issues with Popular Solutions

This solution is well designed and tested¹, whereas there are problems with even the most popular solutions.

Portability issues due to changing pip command line features
Crashing of xargs because common pip or pip3 child process failures
Crowded logging from the raw xargs output
Relying on a Python-to-OS bridge while potentially upgrading it³

The above command uses the simplest and most portable pip syntax in combination with sed and sh to overcome these issues completely. Details of sed operation can be scrutinized with the commented version².

Details

[1] Tested and regularly used in a Linux 4.8.16-200.fc24.x86_64 cluster and tested on five other Linux/Unix flavors. It also runs on Cygwin64 installed on Windows 10. Testing on iOS is needed.

[2] To see the anatomy of the command more clearly, this is the exact equivalent of the above pip3 command with comments:

# match lines from pip's local package list output
# that meet the following three criteria and pass the
# package name to the replacement string in group 1.
# (a) Do not start with invalid characters
# (b) Follow the rule of no white space in the package names
# (c) Immediately follow the package name with an equal sign
sed="s/^([^=# \t\\][^ \t=]*)=.*"

# separate the output of package upgrades with a blank line
sed="$sed/echo"

# indicate what package is being processed
sed="$sed; echo Processing \1 ..."

# perform the upgrade using just the valid package name
sed="$sed; pip3 install -U \1"

# output the commands
sed="$sed/p"

# stream edit the list as above
# and pass the commands to a shell
pip3 freeze --local |sed -rn "$sed" |sh

[3] Upgrading a Python or PIP component that is also used in the upgrading of a Python or PIP component can be a potential cause of a deadlock or package database corruption.

More Robust Solution

For pip3 use this:

pip3 freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip3 install -U \1/p' |sh

For pip, just remove the 3s as such:

pip freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip install -U \1/p' |sh

Solving Issues with Popular Solutions

This solution is well designed and tested¹, whereas there are problems with even the most popular solutions.

Portability issues due to changing pip command line features
Crashing of xargs because common pip or pip3 child process failures
Crowded logging from the raw xargs output
Relying on a Python-to-OS bridge while potentially upgrading it³

The above command uses the simplest and most portable pip syntax in combination with sed and sh to overcome these issues completely. Details of sed operation can be scrutinized with the commented version².

Details

[1] Tested and regularly used in a Linux 4.8.16-200.fc24.x86_64 cluster and tested on five other Linux/Unix flavors. It also runs on Cygwin64 installed on Windows 10. Testing on iOS is needed.

[2] To see the anatomy of the command more clearly, this is the exact equivalent of the above pip3 command with comments:

# match lines from pip's local package list output
# that meet the following three criteria and pass the
# package name to the replacement string in group 1.
# (a) Do not start with invalid characters
# (b) Follow the rule of no white space in the package names
# (c) Immediately follow the package name with an equal sign
sed="s/^([^=# \t\\][^ \t=]*)=.*"

# separate the output of package upgrades with a blank line
sed="$sed/echo"

# indicate what package is being processed
sed="$sed; echo Processing \1 ..."

# perform the upgrade using just the valid package name
sed="$sed; pip3 install -U \1"

# output the commands
sed="$sed/p"

# stream edit the list as above
# and pass the commands to a shell
pip3 freeze --local |sed -rn "$sed" |sh

[3] Upgrading a Python or PIP component that is also used in the upgrading of a Python or PIP component can be a potential cause of a deadlock or package database corruption.

More Robust Solution

For pip3 use this:

pip3 freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip3 install -U \1/p' |sh

For pip, just remove the 3s as such:

pip freeze --local |sed -rn 's/^([^=# \t\\][^ \t=]*)=.*/echo; echo Processing \1 ...; pip install -U \1/p' |sh

OSX Oddity

OSX, as of July 2017, ships with a very old version of sed (a dozen years old). To get extended regular expressions, use -E instead of -r in the solution above.

Solving Issues with Popular Solutions

This solution is well designed and tested¹, whereas there are problems with even the most popular solutions.

Portability issues due to changing pip command line features
Crashing of xargs because common pip or pip3 child process failures
Crowded logging from the raw xargs output
Relying on a Python-to-OS bridge while potentially upgrading it³

The above command uses the simplest and most portable pip syntax in combination with sed and sh to overcome these issues completely. Details of sed operation can be scrutinized with the commented version².

Details

[1] Tested and regularly used in a Linux 4.8.16-200.fc24.x86_64 cluster and tested on five other Linux/Unix flavors. It also runs on Cygwin64 installed on Windows 10. Testing on iOS is needed.

[2] To see the anatomy of the command more clearly, this is the exact equivalent of the above pip3 command with comments:

# match lines from pip's local package list output
# that meet the following three criteria and pass the
# package name to the replacement string in group 1.
# (a) Do not start with invalid characters
# (b) Follow the rule of no white space in the package names
# (c) Immediately follow the package name with an equal sign
sed="s/^([^=# \t\\][^ \t=]*)=.*"

# separate the output of package upgrades with a blank line
sed="$sed/echo"

# indicate what package is being processed
sed="$sed; echo Processing \1 ..."

# perform the upgrade using just the valid package name
sed="$sed; pip3 install -U \1"

# output the commands
sed="$sed/p"

# stream edit the list as above
# and pass the commands to a shell
pip3 freeze --local |sed -rn "$sed" |sh

[3] Upgrading a Python or PIP component that is also used in the upgrading of a Python or PIP component can be a potential cause of a deadlock or package database corruption.

add additional deadlock or race condition issue with using Python to upgrade Python

Source Link

edited Feb 20, 2017 at 0:51

Douglas Daseeco

3.7k
25
29

Loading

Source Link

answered Feb 20, 2017 at 0:45

Douglas Daseeco

3.7k
25
29

Loading

Collectives™ on Stack Overflow

Return to Answer