Upgrading to Django 3 and Python 3

By Rex Resurreccion Jan 01, 2020

On this topic I would like to share about my recent project of upgrading to Django 3 and Python 3 of one of my Client’s back-end system, coming from a Django 1.11 and Python 2.7 environment.

How To Run Multiple Version Of Python Environment

The first step is to setup a test environment that we can use to test and fix the modules that may break while upgrading to Django 3 and Python 3. We can do this by installing Python 3 in user level.

sudo yum install -y wget make libcurl-devel openssl-devel libffi-devel curl gcc 
curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py
python /tmp/get-pip.py
pip install virtualenv
wget https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
tar -xvf Python-3.7.1.tar.xz
mkdir $HOME/.python3
cd Python-3.7.1/
./configure --prefix=$HOME/.python3 --enable-optimizations 
make
make install

After the installation, we can start a new virtual environment with Python 3 interpreter.

virtualenv env --python=$HOME/.python3/bin/python3
source env/bin/activate

Installing Library Dependencies Using Pip Package Manager

The second step is to install the dependencies in requirements.txt. If you need to create your own requirements.txt file in your system, you can use the pip package manager.

pip freeze > requirements.txt

We can also use pip to recursively install all the dependencies, but in my case I end up doing this one at a time to make sure all libraries are working properly.

One of the library that I had issue installing was PyCurl, every time I install this library I would encounter an SSL back-end error. Another one is Django 3, this package would complain about the version of SQLite in my system.

Again you can recursively install all the libraries if you prefer to do so by running the command below.

pip install -r requirements.txt

How To Upgrade SQLite3 From Source

In Django 3, the default database is SQLite3.

sudo su root
wget -P /tmp/ https://www.sqlite.org/2019/sqlite-autoconf-3300100.tar.gz
cd /tmp
tar -zxvf sqlite-autoconf-3300100.tar.gz
cd sqlite-autoconf-3300100.tar.gz
./configure
make 
make install
which sqlite3
/usr/bin/sqlite3 --version
/usr/local/bin/sqlite3 --version
mv -v /usr/bin/sqlite3 /usr/bin/sqlite3.7
cp -v /usr/local/bin/sqlite3 /usr/bin/sqlite3
exit

In here, we downloaded a tarball to install a new version of sqlite3 from source. Notice also that we did not removed the old version, we only moved it to /usr/bin/sqlite3.7, that way we are able to revert back the changes if needed.

How To Fix: PyCurl SSL Backend Error

ssl backend (openssl) is different from compile-time ssl backend (none/other)

To fix the SSL back-end error in PyCurl, you will have to specify the proper SSL library that you have in your system.

pip uninstall pycurl
export PYCURL_SSL_LIBRARY=openssl
pip install pycurl --global-option="--with-openssl"

How To Fix: Django SQLite Version

“django.core.exceptions.improperlyconfigured: sqlite 3.8.3 or later is required”

To solve this issue, install the required version of SQLite3. Follow the instructions in How To Upgrade SQLite3 From Source.

How To Fix: Django Deprecated RequestContext

“TypeError context must be a dict rather than RequestContext”

Passing RequestContext instance to HttpResponse as parameter has been deprecated, even prior to Django 2. Replace this with django.shortcuts.render to render the templates and pass a dict in context parameter instead of a RequestContext instance.

Django 3

from django.shortcuts import render
from django.views import View

class Index(View):
    template_name = ‘index.html’

   def get(self, request):
     context = {
         ‘form’: form,
     }
     return render(request, 'form_template.html', context={'form': form}, content_type=’text/html’)

How To Fix: Django Deprecated urlresolvers

“ImportError: No module named ‘django.core.urlresolvers’ “

django.core.urlresolvers module has been removed since Django 2.0. Change your imports to use django.urls instead.

Django 3

from django.urls import reverse

How To Fix: Urllib Missing Attributes

Some of the modules in urllib has been moved around. Change your imports to use urllib.parse and urllib.request to import quote, quote_plus, urlencode and urlopen.

“AttributeError: module ‘urllib’ has no attribute ‘quote’ “

“AttributeError: module ‘urllib’ has no attribute ‘urlopen’ “

Python 3

from urllib.parse import quote_plus, quote, urlencode
from urllib.request import urlopen

How To Fix: Python String Data Type

A textual data in Python 3 including a Unicode has now a type “str” and Binary data is a type “byte”.

Python 2.7

>>> type(u'Hello World')
<type 'unicode'>
>>> type(r'Hello World')
<type 'str'>
>>> isinstance(u'Hello World', unicode)
True
>>> isinstance(r'Hello World', str)
True

Python 3

>>> type(u'Hello World')
<type 'str'>
>>> type(r'Hello World')
<type 'str'>
>>> isinstance(u'Hello World', str)
True

How To Fix: Python Dict Changed In A Loop

“RuntimeError: dictionary changed size during iteration “

This error happens when you are trying to update the data inside a dictionary while in a loop.

Python 2

>>> staff = {'john': 'Dev', 'doe': 'System Admin'}
>>> for i in staff.keys():
...   if i == 'john':
...     staff[i] = 'Senior Dev'
...
>>> staff
{'john': 'Senior Dev', 'doe': 'System Admin'}

Python 3

>>> staff = {'john': 'Dev', 'doe': 'System Admin'}
>>> for i in list(staff):
...   if i == 'john':
...     staff[i] = 'Senior Dev'
...
>>> staff
{'john': 'Senior Dev', 'doe': 'System Admin'}

How To Fix: Python Deprecated iteritems

“‘dict’ object has no attribute iteritems “

Python 2

>>> staff.iteritems()
<dictionary-itemiterator object at 0x7f170226f260>
>>> staff.viewitems()
dict_items([('john', 'Dev'), ('doe', 'System Admin')])

Python 3

>>> staff.iteritems()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'iteritems'
>>> staff.items()
dict_items([('john', 'Dev'), ('doe', 'System Admin')])

Notice that the dict object in Python 3 no longer have dict.iteritems(). And dict.items() returns an iterator similar to dict.viewitems() in Python 2.7.

How To Fix: Python Deprecated sort

“AttributeError: ‘dict_keys’ objects has no attribute ‘sort'”

Python 2.7

>>> staff_names = staff.keys()
>>> staff_names.sort()
>>> staff_names
['doe', 'john']

Python 3

>>> staff_names = sorted(staff.keys())
>>> staff_names
['doe', 'john']

Notice that the dict_keys object in Python 3 no longer have dict_keys.sort(). To fix this, use another built-in function sorted().

How To Fix: Subprocess Bytes Stream, UnicodeDecodeError

The subprocess module allows you to spawn new processes, connect to input/output/error pipes, and obtain their return codes.

Python 2.7

>>> from subprocess import Popen, STDOUT, PIPE
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT)
>>> type(result.stdout.read())
<type 'str'>

Python 3

In Python 3 the stdin, stdout and stderr has a bytes stream and returns a bytes object by default. Therefore, you cannot make comparison of a regular string into a bytes. However, In subprocess you can set the parameters encoding to unicode or universal_newlines to True to become a text stream for backwards compatibility with Python 2.7 code.

>>> from subprocess import Popen, STDOUT, PIPE
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT)
>>> type(result.stdout.read())
<class 'bytes'>
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT,encoding='utf-8')
>>> type(result.stdout.read())
<class 'str'>
>>> result = Popen(["whoami"],stdout=PIPE,stderr=STDOUT,universal_newlines=True)
>>> type(result.stdout.read())
<class 'str'>

If you do not intend to modify the encoding and the universal_newlines parameters you can also decode the output instead into a regular text stream. Also when passing additional input parameters make sure that the data is converted into bytes.

>>> from subprocess import Popen, STDOUT, PIPE
>>> import json
>>> staff = json.dumps({'john': 'Dev', 'doe': 'System Admin'})
>>> staff_to_bytes = bytes(staff.encode('utf-8'))
>>> staff_to_bytes
b'{"john": "Dev", "doe": "System Admin"}'
>>> p = Popen(["somescript.sh"], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
>>> (result, error) = p.communicate(input=staff_to_bytes)
>>> if int(p.returncode) != 0:
>>>     result = result.decode('utf-8')
>>>     return result

How To Fix: Text to Bytes

“TypeError: a bytes-like object is required, not ‘str'”

>>> haystack = b'bytes stream'
>>> needle = 'bytes'
>>> needle in haystack
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: a bytes-like object is required, not 'str'
>>> bytes(needle.encode('utf-8')) in haystack
True

“TypeError: string argument without an encoding”

>>> bytes('to bytes')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: string argument without an encoding
>>> bytes('to bytes'.encode('utf-8'))
b'to bytes'

End Of Tutorial

Upgrading to Django 3 and Python 3 can become challenging, but if you try to understand the errors, the solution sometimes is relatively simple and if you are lucky you probably just have to upgrade the libraries or use the updated syntax in Python 3.

© YippeeCode.com 2020