Upgrading to Django 3 and Python 3

By Rex Resurreccion Jan 01, 2020

On this topic I would like to share about my recent project of Upgrading to Django 3 and Python 3, coming from a Django 1.11 and Python 2.7 environment.

How To Run Multiple Version Of Python Environment

The first step is to setup a test environment where we can fix the modules that may break while upgrading to Django 3 and Python 3.

Install Python 3 in user level.

sudo yum install -y wget make libcurl-devel openssl-devel libffi-devel curl gcc 
curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py
python /tmp/get-pip.py
pip install virtualenv
wget https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
tar -xvf Python-3.7.1.tar.xz
mkdir $HOME/.python3
cd Python-3.7.1/
./configure --prefix=$HOME/.python3 --enable-optimizations 
make
make install

Start a new virtual environment with Python 3 interpreter.

virtualenv env --python=$HOME/.python3/bin/python3
source env/bin/activate

Installing Library Dependencies Using Pip Package Manager

The second step is to install libraries requirements.txt. Save your own list of Python libraries required in your system.

pip freeze > requirements.txt

I can recursively install all the libraries using pip, but on my end I prefer doing this one at a time making sure that all library dependencies are working properly. To give an example, the project that I migrated is using PyCurl, everytime I install this library I would encounter an ssl backend error. Another one is Django 3 complaining about the version of SQLite in my system.

Again you can recursively install all the libraries if you prefer to do so by running the command below.

pip install -r requirements.txt

How To Upgrade SQLite3 From Source

In Django 3, the default database is SQLite3.

sudo su root
wget -P /tmp/ https://www.sqlite.org/2019/sqlite-autoconf-3300100.tar.gz
cd /tmp
tar -zxvf sqlite-autoconf-3300100.tar.gz
cd sqlite-autoconf-3300100.tar.gz
./configure
make 
make install
which sqlite3
/usr/bin/sqlite3 --version
/usr/local/bin/sqlite3 --version
mv -v /usr/bin/sqlite3 /usr/bin/sqlite3.7
cp -v /usr/local/bin/sqlite3 /usr/bin/sqlite3
exit

Access the new version of sqlite3 in your account. We also moved the old version into /usr/bin/sqlite3.7 so we are able to revert back if needed.

How To Fix: PyCurl SSL Backend Error

ssl backend (openssl) is different from compile-time ssl backend (none/other)

Couple years ago I posted something in Stack Overflow and also the commands in YippeeCode on how to fix the SSL backend error in PyCurl.

pip uninstall pycurl
export PYCURL_SSL_LIBRARY=openssl
pip install pycurl --global-option="--with-openssl"

How To Fix: Django SQLite Version

“django.core.exceptions.improperlyconfigured: sqlite 3.8.3 or later is required”

To solve this issue, install the required version of SQLite3. Follow the instructions in How To Upgrade SQLite3 From Source.

How To Fix: Django Deprecated RequestContext

“TypeError context must be a dict rather than RequestContext”

Passing RequestContext instance to HttpResponse as parameter has been deprecated, even prior to Django 2. Replace this with django.shortcuts.render to render the templates and pass a dict in context parameter instead of RequestContext instance.

Django 3

from django.shortcuts import render
from django.views import View

class Index(View):
    template_name = ‘index.html’

   def get(self, request):
     context = {
         ‘form’: form,
     }
     return render(request, 'form_template.html', context={'form': form}, content_type=’text/html’)

How To Fix: Django Deprecated urlresolvers

“ImportError: No module named ‘django.core.urlresolvers’ “

django.core.urlresolvers module has been removed since Django 2.0. Change your imports to use django.urls instead.

Django 3

from django.urls import reverse

How To Fix: Urllib Missing Attributes

Some of the modules in urllib has been moved around. Change your imports to use urllib.parse and urllib.request to import quote, quote_plus, urlencode and urlopen.

“AttributeError: module ‘urllib’ has no attribute ‘quote’ “

“AttributeError: module ‘urllib’ has no attribute ‘urlopen’ “

Python 3

from urllib.parse import quote_plus, quote, urlencode
from urllib.request import urlopen

How To Fix: Python String Data Type

A textual data in Python 3 including a Unicode has now a type “str” and Binary data is a type “byte”.

Python 2.7

>>> type(u'Hello World')
<type 'unicode'>
>>> type(r'Hello World')
<type 'str'>
>>> isinstance(u'Hello World', unicode)
True
>>> isinstance(r'Hello World', str)
True

Python 3

>>> type(u'Hello World')
<type 'str'>
>>> type(r'Hello World')
<type 'str'>
>>> isinstance(u'Hello World', str)
True

How To Fix: Python Dict Changed In A Loop

“RuntimeError: dictionary changed size during iteration “

This error happens when you are trying to update the data inside a dictionary while on a loop.

Python 2

>>> staff = {'john': 'Dev', 'doe': 'System Admin'}
>>> for i in staff.keys():
...   if i == 'john':
...     staff[i] = 'Senior Dev'
...
>>> staff
{'john': 'Senior Dev', 'doe': 'System Admin'}

Python 3

>>> staff = {'john': 'Dev', 'doe': 'System Admin'}
>>> for i in list(staff):
...   if i == 'john':
...     staff[i] = 'Senior Dev'
...
>>> staff
{'john': 'Senior Dev', 'doe': 'System Admin'}

How To Fix: Python Deprecated iteritems

“‘dict’ object has no attribute iteritems “

Python 2

>>> staff.iteritems()
<dictionary-itemiterator object at 0x7f170226f260>
>>> staff.viewitems()
dict_items([('john', 'Dev'), ('doe', 'System Admin')])

Python 3

>>> staff.iteritems()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'iteritems'
>>> staff.items()
dict_items([('john', 'Dev'), ('doe', 'System Admin')])

Notice that the dict object in Python 3 no longer have dict.iteritems(). And dict.items() returns an iterator similar to dict.viewitems() in Python 2.7.

How To Fix: Python Deprecated sort

“AttributeError: ‘dict_keys’ objects has no attribute ‘sort'”

Python 2.7

>>> staff_names = staff.keys()
>>> staff_names.sort()
>>> staff_names
['doe', 'john']

Python 3

>>> staff_names = sorted(staff.keys())
>>> staff_names
['doe', 'john']

Notice that the dict_keys object in Python 3 no longer have dict_keys.sort(). To fix this, use another built-in module sorted().

How To Fix: Subprocess Bytes Stream, UnicodeDecodeError

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.

Python 2.7

>>> from subprocess import Popen, STDOUT, PIPE
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT)
>>> type(result.stdout.read())
<type 'str'>

Python 3

In Python 3 the stdin, stdout and stderr has a bytes stream and returns a bytes object by default. Therefore, you cannot make comparison of a regular string into a bytes. However, In subprocess you can set the parameters encoding to unicode or universal_newlines to True to become a text stream for backwards compatibility with Python 2.7 code.

>>> from subprocess import Popen, STDOUT, PIPE
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT)
>>> type(result.stdout.read())
<class 'bytes'>
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT,encoding='utf-8')
>>> type(result.stdout.read())
<class 'str'>
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT,universal_newlines=True)
>>> type(result.stdout.read())
<class 'str'>

If you do not intend to modify the encoding and the universal_newlines parameters you can also decode the output instead into a regular text stream. Also when passing additional input parameters make sure that the data is converted to bytes.

>>> from subprocess import Popen, STDOUT, PIPE
>>> import json
>>> staff = json.dumps({'john': 'Dev', 'doe': 'System Admin'})
>>> staff_to_bytes = bytes(staff.encode('utf-8'))
>>> staff_to_bytes
b'{"john": "Dev", "doe": "System Admin"}'
>>> p = Popen(["somescript.sh"], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
>>> (result, error) = p.communicate(input=staff_to_bytes)
>>> if int(p.returncode) != 0:
>>>     result = result.decode('utf-8')
>>>     return result

How To Fix: Text to Bytes

“TypeError: a bytes-like object is required, not ‘str'”

>>> haystack = b'bytes stream'
>>> needle = 'bytes'
>>> needle in haystack
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: a bytes-like object is required, not 'str'
>>> bytes(needle.encode('utf-8')) in haystack
True

“TypeError: string argument without an encoding”

>>> bytes('to bytes')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: string argument without an encoding
>>> bytes('to bytes'.encode('utf-8'))
b'to bytes'

End Of Tutorial

Upgrading to Django 3 and Python 3 could be a challenge, but if you try to understand the error, the solution is relatively simple. I hope this will help other developers that will be doing a migration as well.

Leave a Reply

Your email address will not be published. Required fields are marked *

© YippeeCode.com 2020