Chapter 12 Additional Tips and Tricks
This book has covered everything you need to know to create web applications with Flask. But there's still a lot to discover on your own. In the last chapter, we will cover additional subsections that can be added to the application if necessary.
This chapter will cover the following:
- Full text search with Whoosh
- Full Text Search with Elasticsearch
- use signals
- use cache
- E-mail support for Flask applications
- Understanding asynchronous operations
- Using Celery
introduce
In this chapter, we will first learn how to use Whoosh and Elasticsearch for full text search. Full-text search is very important for web applications such as e-commerce sites that provide a lot of content and options. Next we'll catch signals, which are sent when certain operations are performed in the application. Then implement caching for our Flask app.
We will also see how the application supports sending e-mail. Then you'll see how to implement application asynchrony. Typically, WSGI applications are synchronous and blocking and cannot handle multiple synchronous requests at the same time. We will see how to solve this problem with a simple example. We will also integrate Celery into our application to see the benefits of a task queue to the application.
Full text search with Whoosh
Whoosh is a fast full-text indexing and searching library done in Python. It is a fully Pythonic API that makes it easy and efficient for developers to add search functionality to their applications. In this section, we'll use a package called Flask-WhooshAlchemy, which integrates Whoosh text search functionality with SQLAlchemy for use in Flask applications.
Prepare
Install Flask-WhooshAlchemy with the following command:
$ pip install flask_whooshalchemy
It will install the required packages and dependencies.
Translator's Note
flask_whooshalchemy does not support Python3, and also does not support Chinese well. It is not recommended to use it. You can use jieba.
How to do it
Integrating Whoosh and Flask with SQLAlchemy is very simple. First, we need to provide a path to the Whoosh directory where the model index will be created. This should be done in the application configuration, i.e my_app/__init__.py
.:
app.config['WHOOSH_BASE'] = '/tmp/whoosh'
You can choose any path you like, either absolute or relative.
Next, we need to change the models.py file to make some string/text fields searchable:
import flask.ext.whooshalchemy as whooshalchemy
from my_app import app
class Product(db.Model):
__searchable__ = ['name', 'company']
# … Rest of code as before … #
whooshalchemy.whoosh_index(app, Product)
class Category(db.Model):
__searchable__ = ['name']
# … Rest of code as before … #
whooshalchemy.whoosh_index(app, Category)
Note the __searchable__
statements added by each model. It tells Whoosh to create an index on these fields. Remember that these fields should be of type text or string. The whoosh_index statement tells the application to index these models if they don't already exist.
After doing this, add a handler to search using Whoosh. These can be handled in views.py:
@catalog.route('/product-search-whoosh')
@catalog.route('/product-search-whoosh/<int:page>')
def product_search_whoosh(page=1):
q = request.args.get('q')
products = Product.query.whoosh_search(q)
return render_template(
'products.html', products=products.paginate(page, 10)
)
Here, get the URL parameter via q and pass its value to the whoosh_search() method. This method will perform a full text search on the name and company fields of the Product model. We've set it up earlier to make the name and company in the model searchable.
principle
In Chapter 4, the SQL-based search section, we implemented a basic field-based search method. But with Whoosh we don't need to specify any fields when searching. We can enter any field, and if it matches a searchable field, the results will be returned, sorted by relevance.
First, create some items in the app. Now, open http://127.0.0.1:5000/product-search-whoosh?q=iPhone
, and the results page will display a list of items that contain iPhones.
hint
Whoosh provides some advanced options, we can control which fields can be searched or how the results are sorted. You can explore on your own based on the needs of your app.
other
- refer to
https://pythonhosted.org/Whoosh/
- refer to
https://pypi.python.org/pypi/Flask-WhooshAlchemy
Full Text Search with Elasticsearch
Elasticsearch is a Lucene-based search service, an open source information retrieval library. ElasticSearch provides a distributed full-text search engine with a RESTful web interface and schema-free JSON documents. In this section, we will use Elasticsearch to perform full-text search for our Flask application.
Prepare
We'll use a Python library called pyelasticsearch, which makes working with Elasticsearch easy:
$ pip install pyelasticsearch
We also need to install the Elasticsearch service itself. can be http://www.elasticsearch.org/download/
downloaded from. Unzip the file and run the following command:
$ bin/elasticsearch
By default, the http://localhost:9200/
Elasticsearch service will be running on .
How to do it
To demonstrate the integration, we will start by adding Elasticsearch to the application configuration, namely my_app/__init__.py
:
from pyelasticsearch import ElasticSearch
from pyelasticsearch.exceptions import IndexAlreadyExistsError
es = ElasticSearch('http://localhost:9200/')
try:
es.create_index('catalog')
except IndexAlreadyExistsError, e:
pass
Here, we created an es object from the ElasticSearch class, which received the server URL. Then an index called catalog is created. They are handled in a try-except block because an IndexAlradyExistsError will be thrown if the index already exists, which can be ignored by catching the exception.
Next, we need to add documents to the Elasticsearch index. This can be done in the view and model, but it is best to add it at the model layer. So, we will do this in models.py:
from my_app import es
class Product(db.Model):
def add_index_to_es(self):
es.index('catalog', 'product', {
'name': self.name,
'category': self.category.name
})
es.refresh('catalog')
class Category(db.Model):
def add_index_to_es(self):
es.index('catalog', 'category', {
'name': self.name,
})
es.refresh('catalog')
Here, in each model, we add a method called add_index_to_es(), which will add the document corresponding to the current Product or Category object to the catalog index, along with the associated document type, i.e. product or category. Finally, we refresh the index to ensure that the newly created index can be searched.
add_index_to_es()
Methods can be called when we create, update, or delete items. To demonstrate, we just add this method when creating the item in views.py:
from my_app import es
def create_product():
#... normal product creation as always ...#
db.session.commit()
product.add_index_to_es()
#... normal process as always ...#
@catalog.route('/product-search-es')
@catalog.route('/product-search-es/<int:page>')
def product_search_es(page=1):
q = request.args.get('q')
products = es.search(q)
return products
Also, add a product_search_es()
method that allows searching on the Elasticsearch you just created. Do the same for the create_category() method.
How to do it
Assume that some items have been created in each category. Now, if we open http://127.0.0.1:5000/product-search-es?q=galaxy
it, we will see a response similar to the screenshot below:
use signals
Signals can be understood as events that occur in the application. These events can be subscribed to by some specific receiver, which will trigger a function when the event occurs. The occurrence of the event is broadcast by the sender, and the sender can specify the parameters that can be used in the receiver trigger function.
hint
You should avoid modifying any application data in signals, because signals are not executed in the order specified and can easily lead to data corruption.
Prepare
We'll use a Python library called binker, which provides some signaling features. Flask has built-in support for blinkers and can use signals to a great extent. Some of these core signals are provided by Flask.
In this section, we will use the application from the previous section to add additional product and category documents to the index via signals.
How to do it
First, we create the signals when new items and categories are created. This can be handled in models.py. It can also be handled in any file we wish, since signals are created at the global scope:
from blinker import Namespace
catalog_signals = Namespace()
product_created = catalog_signals.signal('product-created')
category_created = catalog_signals.signal('category-created')
We use Namespace to create signals, because this will create them in the custom namespace instead of the global space, which will help manage signals. We created two signals that can be understood from their names.
After that, we need to create subscribers for these signals and bind functions to them. For this purpose, add_index_to_es()
it needs to be removed and a new function needs to be created in the global scope:
def add_product_index_to_es(sender, product):
es.index('catalog', 'product', {
'name': product.name,
'category': product.category.name
})
es.refresh('catalog')
product_created.connect(add_product_index_to_es, app)
def add_category_index_to_es(sender, category):
es.index('catalog', 'category', {
'name': category.name,
})
es.refresh('catalog')
category_created.connect(add_category_index_to_es, app)
In the previous code, we used .connect() to create a subscriber for the signal. This method accepts a function that will be called when the event occurs. It also accepts a sender as an optional parameter. The app object is provided as the sender, because we don't want our function to be called whenever it fires anywhere in the app. This is especially true in the case of extensions, which can be used by multiple applications. The function called by the receiver receives the sender as the first parameter, which is usually None if not provided by the sender. We provide it product/category
as the second parameter in order to add this record to the Elasticsearch index.
Now, we just need to trigger a signal that can be caught by the receiver. These can be handled in views.py. To do this, remove the add_index_to_es()
methods and replace them with the .send() method:
from my_app.catalog.models import product_created, category_created
def create_product():
#... normal product creation as always ...#
db.session.commit()
product_created.send(app, product=product)
# product.add_index_to_es()
#... normal process as always ...#
Do the same for create_category().
principle
When an item is created, the product_created
signal is triggered, the app is the sender, and the item is the key parameter. This will be captured in models.py and when the add_product_index_to_es()
function is called, the documentation will be added to the directory index.
other
- References
https://pypi.python.org/pypi/blinker
- References
http://flask.pocoo.org/docs/0.10/signals/#core-signals
- Signals provided by Flask-SQLAlchemy can be
https://pythonhosted.org/Flask-SQLAlchemy/signals.html
found at
use cache
Caching becomes an essential part of any web application when increased application response time becomes an issue. Flask itself doesn't provide any caching support by default, but Werkzeug does. Werkzeug provides basic support for caching and can use various backends, such as Memcached and Redis.
Prepare
We'll install an extension called Flask-Cache, which will greatly simplify the use of the cache:
$ pip install Flask-Cache
How to do it
First, the Cache needs to be initialized. will be processed in the application configuration, my_app/__init__.py
:
from flask.ext.cache import Cache
cache = Cache(app, config={'CACHE_TYPE': 'simple'})
Here, simple is used as the type of Cache, and the cache is stored in memory. This is not recommended in a production environment. For production environments, use Redis, Memcached, file systems, etc. Flask-Cache supports all of these.
Next, add the cache to the method; this is very easy to do. We just need to add @cache.cached(timeout=
from my_app import cache
@catalog.route('/categories')
@cache.cached(timeout=120)
def categories():
# Fetch and display the list of categories
This caching method is stored in the form of key-value pairs, where the key is the request path and the value is the output value of this method.
principle
After adding the preceding code, check that the cache is in effect. First visit to http://127.0.0.1:5000/categories
get a list of categories. will store a key-value pair for this URL in the cache. Now, quickly create a new category and visit the product category listing page again. You will see that the newly created category is not listed. Wait a few minutes, then refresh the page. New categories will be displayed. This is because the category list is cached when it is accessed for the first time, and the effective time is 2 minutes, or 120 seconds.
This may seem like a bug in the application, but in large applications, reducing the number of accesses to the database is a boon and the overall application experience improves. Caches are typically used for handlers whose results are not updated frequently.
More
Many of us may think that within a single category or product page, this kind of caching will fail because there is a separate page for each record. The solution to this problem is memoization. When a method is called multiple times with the same parameters, the result should be loaded from the cache instead of accessing the database. Implementing memoization is very simple:
@catalog.route('/product/<id>')
@cache.memoize(120)
def product(id):
# Fetch and display the product
Now, if we type in the browser http://127.0.0.1:5000/product/1
, the first request will read data from the database. However, the second time, if the same access is done, the data will be loaded from the cache. If we open another product page http://127.0.0.1:5000/product/2
, the product details will be fetched from the database.
other
- To learn more about Flask-Cache, see
https://pythonhosted.org/Flask-Cache/
- For more memoization, see
http://en.wikipedia.org/wiki/Memoization
E-mail support for Flask applications
Sending emails is one of the most basic functions of any web application. Python-based applications can use smptblib to accomplish this very easily. In Flask, using the Flask-Mail extension simplifies this process even more.
Prepare
Flask-Mail is installed via pip:
$ pip install Flask-Mail
Let's take a simple example, whenever a new category is added, an e-mail will be sent to the application manager.
How to do it
First, the Mail object needs to be instantiated in the application configuration, namely my_app/__init__.py
:
from flask_mail import Mail
app.config['MAIL_SERVER'] = 'smtp.gmail.com'
app.config['MAIL_PORT'] = 587
app.config['MAIL_USE_TLS'] = True
app.config['MAIL_USERNAME'] = 'gmail_username'
app.config['MAIL_PASSWORD'] = 'gmail_password'
app.config['MAIL_DEFAULT_SENDER'] = ('Sender name', 'sender email')
mail = Mail(app)
Also, we need some configuration to create the e-mail service and sender account. The preceding code is a simple example of configuring a Gmail account. Any SMTP service can be set up like this. There are a few other options to choose from, see the Flask-Mail documentation https://pythonhosted.org/Flask-Mail
.
principle
To send e-mail when the category is created, we need to make the following changes to view.py:
from my_app import mail
from flask_mail import Message
@catalog.route('/category-create', methods=['GET', 'POST'])
def create_category():
# … Create category … #
db.session.commit()
message = Message(
"New category added",
recipients=['[email protected]']
)
message.body = 'New category "%s" has been created' % category.name
mail.send(message)
# … Rest of the process … #
Here, a new e-mail will be sent from the sender we created to the recipient list.
More
Now, suppose you need to send a very large email that contains a lot of HTML text. Writing this all in a Python file would make the code ugly and unmanageable. An easy way to do this is to create a template and render it when sending the email. I have created two templates, one is HTML text and one is plain text.
The category-create-email-text.html template looks like this:
A new category has been added to the catalog.
The name of the category is {{ category.name }}.
Click on the URL below to access the same:
{{ url_for('catalog.category', id=category.id, _external = True) }}
This is an automated email. Do not reply to it.
The category-create-email-html.html template looks like this:
<p>A new category has been added to the catalog.</p>
<p>The name of the category is <a href="{{ url_for('catalog.category', id=category.id, _external = True) }}">
<h2>{{ category.name }}</h2>
</a>.
</p>
<p>This is an automated email. Do not reply to it.</p>
After that, we need to modify the code that created the e-mail in views.py before;
message.body = render_template(
"category-create-email-text.html", category=category
)
message.html = render_template(
"category-create-email-html.html", category=category
)
other
- Read the next section to see how to handle the time-consuming email sending process in an asynchronous thread
Understanding asynchronous operations
There are some operations in the app that can be time-consuming and make the app slower, even if it's not really slow. But this degrades the user experience. To solve this problem, the easiest way is to use threads for asynchronous operations. In this section, we will use Python's thread and threading library to accomplish this function. The threading library is a simple interface to thread; it provides more functionality and hides some things that users don't use very often.
Prepare
We will use the application code from the previous section. Many of us may have noticed that while the email is being sent, the app is waiting for the process to complete, which is in fact unnecessary. E-mail sending can be handled in the background, so that our application becomes responsive to the user.
How to do it
Using the thread library to handle asynchrony is very simple. Just add this code to views.py:
import thread
def send_mail(message):
with app.app_context():
mail.send(message)
# Replace the line below in create_category()
#mail.send(message)
# by
thread.start_new_thread(send_mail, (message,))
As you can see, the mail occurs in a new thread that receives a parameter called message. We need to create a new send_mail()
method, because our e-mail template contains url_for
, so the send_mail
method can only run in the application context, by default it is not available in a newly created thread.
At the same time, the threading library can also be used to send e-mail:
from threading import Thread
# Replace the previously added line in create_category() by
new_thread = Thread(target=send_mail, args=[message])
new_thread.start()
In effect, the effect is the same as before, but the threading library provides the flexibility to start threads when needed, rather than creating and starting threads at the same time.
principle
It is easy to observe the effect of the above code. Compare the performance of this application with the application of the previous section. You will find the app more responsive. Another way is to monitor the log output, the newly created category page will be loaded before the e-mail is sent.
Using Celery
Celery is a task queue for Python. In the early days, there was an extension that integrated Flask and Celery, but in Celery 3.0, this extension was abandoned. Now, Celery can be used directly in Flask with only some configuration. In the previous section, we implemented asynchronous sending of emails, and this section will use Celery to accomplish the same function.
Prepare
Install Celery:
$ pip install celery
To use Celery with Flask, we only need to modify a little Flask app configuration. Here, Redis is used as a broker.
We'll use the application from the previous subsection and then use Celery to complete it.
How to do it
The first thing to do is to do some configuration in the application configuration file, namely my_app/__init__.py
:
from celery import Celery
app.config.update(
CELERY_BROKER_URL='redis://localhost:6379',
CELERY_RESULT_BACKEND='redis://localhost:6379'
)
def make_celery(app):
celery = Celery(
app.import_name, broker=app.config['CELERY_BROKER_URL']
)
celery.conf.update(app.config)
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
with app.app_context():
return TaskBase.__call__(self, *args, **kwargs)
celery.Task = ContextTask
return celery
The preceding code comes directly from the Flask website, and in most cases can be used in an application like this:
celery = make_celery(app)
To run the Celery process, execute the following command:
$ celery worker -b redis://localhost:6379 --app=my_app.celery -l INFO
hint
Make sure Redis is running on the broker URL, as specified in the configuration.
Here, -b specifies the broker, and -app specifies the celery object created in the configuration file.
Now, just use the celery object in views.py to send emails asynchronously:
from my_app import celery
@celery.task()
def send_mail(message):
with app.app_context():
mail.send(message)
# Add this line wherever the email needs to be sent
send_mail.apply_async((message,))
If we want a method to run as a Celery task just add the @celery.task decorator. These methods are automatically detected by the Celery process.
principle
Now that we have an item created and an email sent, we can see a task running in the Celery process log that looks like this:
[2014-08-28 01:16:47,365: INFO/MainProcess] Received task: my_app.catalog.views.send_mail[d2ca07ae-6b47-4b76-9935-17b826cdc340]
[2014-08-28 01:16:55,695: INFO/MainProcess] Task my_app.catalog.views.send_mail[d2ca07ae-6b47-4b76-9935-17b826cdc340] succeeded in 8.329121886s: None
other
- For more information on Celery, see
http://docs.celeryproject.org/en/latest/index.html