Flask Framework Cookbook - Chapter 12 Additional Tips and Tricks

Chapter 12 Additional Tips and Tricks

This book has covered everything you need to know to create web applications with Flask. But there's still a lot to discover on your own. In the last chapter, we will cover additional subsections that can be added to the application if necessary.

This chapter will cover the following:

• Full text search with Whoosh
• Full Text Search with Elasticsearch
• use signals
• use cache
• E-mail support for Flask applications
• Understanding asynchronous operations
• Using Celery

introduce

In this chapter, we will first learn how to use Whoosh and Elasticsearch for full text search. Full-text search is very important for web applications such as e-commerce sites that provide a lot of content and options. Next we'll catch signals, which are sent when certain operations are performed in the application. Then implement caching for our Flask app.
We will also see how the application supports sending e-mail. Then you'll see how to implement application asynchrony. Typically, WSGI applications are synchronous and blocking and cannot handle multiple synchronous requests at the same time. We will see how to solve this problem with a simple example. We will also integrate Celery into our application to see the benefits of a task queue to the application.

Full text search with Whoosh

Whoosh is a fast full-text indexing and searching library done in Python. It is a fully Pythonic API that makes it easy and efficient for developers to add search functionality to their applications. In this section, we'll use a package called Flask-WhooshAlchemy, which integrates Whoosh text search functionality with SQLAlchemy for use in Flask applications.

Prepare

Install Flask-WhooshAlchemy with the following command:

$ pip install flask_whooshalchemy

It will install the required packages and dependencies.

Translator's Note

flask_whooshalchemy does not support Python3, and also does not support Chinese well. It is not recommended to use it. You can use jieba.

How to do it

Integrating Whoosh and Flask with SQLAlchemy is very simple. First, we need to provide a path to the Whoosh directory where the model index will be created. This should be done in the application configuration, i.e my_app/__init__.py.:

app.config['WHOOSH_BASE'] = '/tmp/whoosh'

You can choose any path you like, either absolute or relative.

Next, we need to change the models.py file to make some string/text fields searchable:

import flask.ext.whooshalchemy as whooshalchemy
from my_app import app

class Product(db.Model):
    __searchable__ = ['name', 'company']
    # … Rest of code as before … #

whooshalchemy.whoosh_index(app, Product)

class Category(db.Model):
    __searchable__ = ['name']
    # … Rest of code as before … #

whooshalchemy.whoosh_index(app, Category)

Note the __searchable__statements added by each model. It tells Whoosh to create an index on these fields. Remember that these fields should be of type text or string. The whoosh_index statement tells the application to index these models if they don't already exist.

After doing this, add a handler to search using Whoosh. These can be handled in views.py:

@catalog.route('/product-search-whoosh')
@catalog.route('/product-search-whoosh/<int:page>')
def product_search_whoosh(page=1):
    q = request.args.get('q')
    products = Product.query.whoosh_search(q)
    return render_template(
        'products.html', products=products.paginate(page, 10)
    )

Here, get the URL parameter via q and pass its value to the whoosh_search() method. This method will perform a full text search on the name and company fields of the Product model. We've set it up earlier to make the name and company in the model searchable.

principle

In Chapter 4, the SQL-based search section, we implemented a basic field-based search method. But with Whoosh we don't need to specify any fields when searching. We can enter any field, and if it matches a searchable field, the results will be returned, sorted by relevance.

First, create some items in the app. Now, open http://127.0.0.1:5000/product-search-whoosh?q=iPhone, and the results page will display a list of items that contain iPhones.

hint

Whoosh provides some advanced options, we can control which fields can be searched or how the results are sorted. You can explore on your own based on the needs of your app.

other

• refer tohttps://pythonhosted.org/Whoosh/
• refer tohttps://pypi.python.org/pypi/Flask-WhooshAlchemy

Full Text Search with Elasticsearch

Elasticsearch is a Lucene-based search service, an open source information retrieval library. ElasticSearch provides a distributed full-text search engine with a RESTful web interface and schema-free JSON documents. In this section, we will use Elasticsearch to perform full-text search for our Flask application.

Prepare

We'll use a Python library called pyelasticsearch, which makes working with Elasticsearch easy:

$ pip install pyelasticsearch

We also need to install the Elasticsearch service itself. can be http://www.elasticsearch.org/download/downloaded from. Unzip the file and run the following command:

$ bin/elasticsearch

By default, the http://localhost:9200/Elasticsearch service will be running on .

How to do it

To demonstrate the integration, we will start by adding Elasticsearch to the application configuration, namely my_app/__init__.py:

from pyelasticsearch import ElasticSearch
from pyelasticsearch.exceptions import IndexAlreadyExistsError

es = ElasticSearch('http://localhost:9200/')
try:
    es.create_index('catalog')
except IndexAlreadyExistsError, e:
    pass

Here, we created an es object from the ElasticSearch class, which received the server URL. Then an index called catalog is created. They are handled in a try-except block because an IndexAlradyExistsError will be thrown if the index already exists, which can be ignored by catching the exception.

Next, we need to add documents to the Elasticsearch index. This can be done in the view and model, but it is best to add it at the model layer. So, we will do this in models.py:

from my_app import es

class Product(db.Model):

    def add_index_to_es(self):
        es.index('catalog', 'product', {
            'name': self.name,
            'category': self.category.name
        })
        es.refresh('catalog')

class Category(db.Model):

    def add_index_to_es(self):
        es.index('catalog', 'category', {
            'name': self.name,
        })
        es.refresh('catalog')

Here, in each model, we add a method called add_index_to_es(), which will add the document corresponding to the current Product or Category object to the catalog index, along with the associated document type, i.e. product or category. Finally, we refresh the index to ensure that the newly created index can be searched.

add_index_to_es()Methods can be called when we create, update, or delete items. To demonstrate, we just add this method when creating the item in views.py:

from my_app import es

def create_product():
    #... normal product creation as always ...#
    db.session.commit()
    product.add_index_to_es()
    #... normal process as always ...#

@catalog.route('/product-search-es')
@catalog.route('/product-search-es/<int:page>')
def product_search_es(page=1):
    q = request.args.get('q')
    products = es.search(q)
    return products

Also, add a product_search_es()method that allows searching on the Elasticsearch you just created. Do the same for the create_category() method.

How to do it

Assume that some items have been created in each category. Now, if we open http://127.0.0.1:5000/product-search-es?q=galaxyit, we will see a response similar to the screenshot below:

use signals

Signals can be understood as events that occur in the application. These events can be subscribed to by some specific receiver, which will trigger a function when the event occurs. The occurrence of the event is broadcast by the sender, and the sender can specify the parameters that can be used in the receiver trigger function.

hint

You should avoid modifying any application data in signals, because signals are not executed in the order specified and can easily lead to data corruption.

Prepare

We'll use a Python library called binker, which provides some signaling features. Flask has built-in support for blinkers and can use signals to a great extent. Some of these core signals are provided by Flask.

In this section, we will use the application from the previous section to add additional product and category documents to the index via signals.

How to do it

First, we create the signals when new items and categories are created. This can be handled in models.py. It can also be handled in any file we wish, since signals are created at the global scope:

from blinker import Namespace

catalog_signals = Namespace()
product_created = catalog_signals.signal('product-created')
category_created = catalog_signals.signal('category-created')

We use Namespace to create signals, because this will create them in the custom namespace instead of the global space, which will help manage signals. We created two signals that can be understood from their names.

After that, we need to create subscribers for these signals and bind functions to them. For this purpose, add_index_to_es()it needs to be removed and a new function needs to be created in the global scope:

def add_product_index_to_es(sender, product):
    es.index('catalog', 'product', {
        'name': product.name,
        'category': product.category.name   
    })
    es.refresh('catalog')

product_created.connect(add_product_index_to_es, app)

def add_category_index_to_es(sender, category):
    es.index('catalog', 'category', {
        'name': category.name,
    })
    es.refresh('catalog')

category_created.connect(add_category_index_to_es, app)

In the previous code, we used .connect() to create a subscriber for the signal. This method accepts a function that will be called when the event occurs. It also accepts a sender as an optional parameter. The app object is provided as the sender, because we don't want our function to be called whenever it fires anywhere in the app. This is especially true in the case of extensions, which can be used by multiple applications. The function called by the receiver receives the sender as the first parameter, which is usually None if not provided by the sender. We provide it product/categoryas the second parameter in order to add this record to the Elasticsearch index.

Now, we just need to trigger a signal that can be caught by the receiver. These can be handled in views.py. To do this, remove the add_index_to_es()methods and replace them with the .send() method:

from my_app.catalog.models import product_created, category_created

def create_product():
    #... normal product creation as always ...#
    db.session.commit()
    product_created.send(app, product=product)
    # product.add_index_to_es()
    #... normal process as always ...#

Do the same for create_category().

principle

When an item is created, the product_createdsignal is triggered, the app is the sender, and the item is the key parameter. This will be captured in models.py and when the add_product_index_to_es()function is called, the documentation will be added to the directory index.

other

• Referenceshttps://pypi.python.org/pypi/blinker
• Referenceshttp://flask.pocoo.org/docs/0.10/signals/#core-signals
• Signals provided by Flask-SQLAlchemy can be https://pythonhosted.org/Flask-SQLAlchemy/signals.htmlfound at

use cache

Caching becomes an essential part of any web application when increased application response time becomes an issue. Flask itself doesn't provide any caching support by default, but Werkzeug does. Werkzeug provides basic support for caching and can use various backends, such as Memcached and Redis.

Prepare

We'll install an extension called Flask-Cache, which will greatly simplify the use of the cache:

$ pip install Flask-Cache

How to do it

First, the Cache needs to be initialized. will be processed in the application configuration, my_app/__init__.py:

from flask.ext.cache import Cache

cache = Cache(app, config={'CACHE_TYPE': 'simple'})

Here, simple is used as the type of Cache, and the cache is stored in memory. This is not recommended in a production environment. For production environments, use Redis, Memcached, file systems, etc. Flask-Cache supports all of these.

Next, add the cache to the method; this is very easy to do. We just need to add @cache.cached(timeout=) decorator. Listing all product categories can be done like this (handled in views.py):

from my_app import cache
@catalog.route('/categories')
@cache.cached(timeout=120)
def categories():
    # Fetch and display the list of categories  

This caching method is stored in the form of key-value pairs, where the key is the request path and the value is the output value of this method.

principle

After adding the preceding code, check that the cache is in effect. First visit to http://127.0.0.1:5000/categoriesget a list of categories. will store a key-value pair for this URL in the cache. Now, quickly create a new category and visit the product category listing page again. You will see that the newly created category is not listed. Wait a few minutes, then refresh the page. New categories will be displayed. This is because the category list is cached when it is accessed for the first time, and the effective time is 2 minutes, or 120 seconds.

This may seem like a bug in the application, but in large applications, fewer hits to the database are a boon, and the overall application experience improves. Caches are typically used for handlers whose results are not updated frequently.

More

Many of us may think that within a single category or product page, this kind of caching will fail because there is a separate page for each record. The solution to this problem is memoization. When a method is called multiple times with the same parameters, the result should be loaded from the cache instead of accessing the database. Implementing memoization is very simple:

@catalog.route('/product/<id>')
@cache.memoize(120)
def product(id):
    # Fetch and display the product

Now, if we type in the browser http://127.0.0.1:5000/product/1, the first request will read data from the database. However, the second time, if the same access is done, the data will be loaded from the cache. If we open another product page http://127.0.0.1:5000/product/2, the product details will be fetched from the database.

other

• To learn more about Flask-Cache, seehttps://pythonhosted.org/Flask-Cache/
• For more memoization, seehttp://en.wikipedia.org/wiki/Memoization

E-mail support for Flask applications

Sending emails is one of the most basic functions of any web application. Python-based applications can use smptblib to accomplish this very easily. In Flask, using the Flask-Mail extension simplifies this process even more.

Prepare

Flask-Mail is installed via pip:

$ pip install Flask-Mail

Let's take a simple example, whenever a new category is added, an e-mail will be sent to the application manager.

How to do it

First, the Mail object needs to be instantiated in the application configuration, namely my_app/__init__.py:

from flask_mail import Mail

app.config['MAIL_SERVER'] = 'smtp.gmail.com'
app.config['MAIL_PORT'] = 587
app.config['MAIL_USE_TLS'] = True
app.config['MAIL_USERNAME'] = 'gmail_username'
app.config['MAIL_PASSWORD'] = 'gmail_password'
app.config['MAIL_DEFAULT_SENDER'] = ('Sender name', 'sender email')
mail = Mail(app)

Also, we need some configuration to create the e-mail service and sender account. The preceding code is a simple example of configuring a Gmail account. Any SMTP service can be set up like this. There are a few other options to choose from, see the Flask-Mail documentation https://pythonhosted.org/Flask-Mail.

principle

To send e-mail when the category is created, we need to make the following changes to view.py:

from my_app import mail
from flask_mail import Message

@catalog.route('/category-create', methods=['GET', 'POST'])
def create_category():
    # … Create category … #
    db.session.commit()
    message = Message(
        "New category added",
        recipients=['[email protected]']
    )
    message.body = 'New category "%s" has been created' % category.name
    mail.send(message)
    # … Rest of the process … #

Here, a new e-mail will be sent from the sender we created to the recipient list.

More

Now, suppose you need to send a very large email that contains a lot of HTML text. Writing this all in a Python file would make the code ugly and unmanageable. An easy way to do this is to create a template and render it when sending the email. I have created two templates, one is HTML text and one is plain text.

The category-create-email-text.html template looks like this:

A new category has been added to the catalog.
The name of the category is {{ category.name }}.
Click on the URL below to access the same:
{{ url_for('catalog.category', id=category.id, _external = True) }}
This is an automated email. Do not reply to it.

The category-create-email-html.html template looks like this:

<p>A new category has been added to the catalog.</p>
<p>The name of the category is <a href="{{ url_for('catalog.category', id=category.id, _external = True) }}">
        <h2>{{ category.name }}</h2>
    </a>.
</p>
<p>This is an automated email. Do not reply to it.</p>

After that, we need to modify the code that created the e-mail in views.py before;

message.body = render_template(
    "category-create-email-text.html", category=category
)
message.html = render_template(
    "category-create-email-html.html", category=category
)

other

• Read the next section to see how to handle the time-consuming email sending process in an asynchronous thread

Understanding asynchronous operations

There are some operations in the app that can be time-consuming and make the app slower, even if it's not really slow. But this degrades the user experience. To solve this problem, the easiest way is to use threads for asynchronous operations. In this section, we will use Python's thread and threading library to accomplish this function. The threading library is a simple interface to thread; it provides more functionality and hides some things that users don't use very often.

Prepare

We will use the application code from the previous section. Many of us may have noticed that while the email is being sent, the app is waiting for the process to complete, which is in fact unnecessary. E-mail sending can be handled in the background, so that our application becomes responsive to the user.

How to do it

Using the thread library to handle asynchrony is very simple. Just add this code to views.py:

import thread

def send_mail(message):
    with app.app_context():
        mail.send(message)
# Replace the line below in create_category()
#mail.send(message)
# by
thread.start_new_thread(send_mail, (message,))

As you can see, the mail occurs in a new thread that receives a parameter called message. We need to create a new send_mail()method, because our e-mail template contains url_for, so the send_mailmethod can only run in the application context, by default it is not available in a newly created thread.

At the same time, the threading library can also be used to send e-mail:

from threading import Thread

# Replace the previously added line in create_category() by
new_thread = Thread(target=send_mail, args=[message])
new_thread.start()

In effect, the effect is the same as before, but the threading library provides the flexibility to start threads when needed, rather than creating and starting threads at the same time.

principle

It is easy to observe the effect of the above code. Compare the performance of this application with the application of the previous section. You will find the app more responsive. Another way is to monitor the log output, the newly created category page will be loaded before the e-mail is sent.

Using Celery

Celery is a task queue for Python. In the early days, there was an extension that integrated Flask and Celery, but in Celery 3.0, this extension was abandoned. Now, Celery can be used directly in Flask with only some configuration. In the previous section, we implemented asynchronous sending of emails, and this section will use Celery to accomplish the same function.

Prepare

Install Celery:

$ pip install celery

To use Celery with Flask, we only need to modify a little Flask app configuration. Here, Redis is used as a broker.
We'll use the application from the previous subsection and then use Celery to complete it.

How to do it

The first thing to do is to do some configuration in the application configuration file, namely my_app/__init__.py:

from celery import Celery

app.config.update(
    CELERY_BROKER_URL='redis://localhost:6379',
    CELERY_RESULT_BACKEND='redis://localhost:6379'
)
def make_celery(app):
    celery = Celery(
        app.import_name, broker=app.config['CELERY_BROKER_URL']
    )
    celery.conf.update(app.config)
    TaskBase = celery.Task
    class ContextTask(TaskBase):
        abstract = True
        def __call__(self, *args, **kwargs):
            with app.app_context():
                return TaskBase.__call__(self, *args, **kwargs)
    celery.Task = ContextTask
    return celery

The preceding code comes directly from the Flask website, and in most cases can be used in an application like this:

celery = make_celery(app)

To run the Celery process, execute the following command:

$ celery worker -b redis://localhost:6379 --app=my_app.celery -l INFO

hint

Make sure Redis is running on the broker URL, as specified in the configuration.

Here, -b specifies the broker, and -app specifies the celery object created in the configuration file.
Now, just use the celery object in views.py to send emails asynchronously:

from my_app import celery

@celery.task()
def send_mail(message):
    with app.app_context():
        mail.send(message)

# Add this line wherever the email needs to be sent
send_mail.apply_async((message,))

If we want a method to run as a Celery task just add the @celery.task decorator. These methods are automatically detected by the Celery process.

principle

Now that we have an item created and an email sent, we can see a task running in the Celery process log that looks like this:

[2014-08-28 01:16:47,365: INFO/MainProcess] Received task: my_app.catalog.views.send_mail[d2ca07ae-6b47-4b76-9935-17b826cdc340]
[2014-08-28 01:16:55,695: INFO/MainProcess] Task my_app.catalog.views.send_mail[d2ca07ae-6b47-4b76-9935-17b826cdc340] succeeded in 8.329121886s: None

other

• For more information on Celery, seehttp://docs.celeryproject.org/en/latest/index.html