Generators in Python

Python Generators: Here, we are going to learn about the Python generators with examples, also explain about the generators using list comprehension.
Submitted by Sapna Deraje Radhakrishna, on November 02, 2019

Generators are similar to list comprehensions but are surrounded by parentheses. The generators and list comprehension produce the same result but operate differently.

When a list comprehension executes, it produces all of its data before any other processing occurring. The list comprehension takes a long time to produce data, delays any other code from running until the list comprehension concludes.

For example,

Python 3.6.8 (default, Apr 25 2019, 21:02:35)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> for i in [x*3 for x in [1,2,3,4]]:
...     print(i)
...
3
6
9
12
>>>

In the above example, since the list is small the memory consumed is very less. However, if the list produces an enormous number of items then we have an issue.

Generators produce data items one at a time

The above list comprehension is modified to be a generator as below,

>>> for i in (x*3 for x in [1,2,3,4]):
...     print(i)
...

Unlike in the list comprehension, which must wrap up before any other code can execute, a generator yields data as soon as the data is produced by the generators code, which means if the generator generates enormous data items, and any code that’s waiting to consume the result generated executes immediately.

Example:

Using list comprehension, in the below example the response for both the URLs is returned at same time, which means the print statement waits for the list to perform its action,

Python 3.6.8 (default, Apr 25 2019, 21:02:35)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import requests
>>> from datetime import datetime
>>> urls = ('http://www.includehelp.com', 'https://www.linkedin.com/')
>>> for resp in [requests.get(url) for url in urls]:
...     print(len(resp.content), '->', resp.status_code, '->', resp.url, '->', datetime.now())
...
132281 -> 200 -> https://www.includehelp.com/ -> 2019-10-30 17:51:41.099558
82131 -> 200 -> https://www.linkedin.com/ -> 2019-10-30 17:51:41.099726

Using generators, in the below example the response time for each of URL in the URLs is considerable different and explains the generator execution.

Python 3.6.8 (default, Apr 25 2019, 21:02:35)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import requests
>>> from datetime import datetime
>>> urls = ('http://www.includehelp.com', 'https://www.linkedin.com/')
>>> for resp in (requests.get(url) for url in urls):
...     print(len(resp.content), '->', resp.status_code, '->', resp.url, '->', datetime.now())
...
132281 -> 200 -> https://www.includehelp.com/ -> 2019-10-30 17:51:54.832263
82124 -> 200 -> https://www.linkedin.com/ -> 2019-10-30 17:51:54.971670