从0学DRF（实战与源码剖析）——节流

介绍

当服务的接口被频繁调用，我们可以使用负载均衡的策略，而DRF内部为我们实现了节流（访问频率控制的功能）

节流内部原理概述

DRF框架在我们访问的时候，会记录来访者的IP，会记录到字典里边作为Key ，而把访问的时间加入到一个列表里边作为字典的值。

节流实现的算法

面试问：自定义节流有思路吗？

额，比如以5秒内只能访问3次为例，首先用户访问后，它的IP会作为字典的key，然后它的访问时间加入到字典里边作为字典的值，当下一次访问的时候，先判断一下，如果列表里边的最后一个元素的时间比当前时间超过了5秒，就把它从列表里面弹出，然后看倒数第二个元素，如果也超过5秒，也把它弹出，以此类推，如果列表里的元素个数大于或者等于3，那么那个请求就不能让它加入到列表里边，否则，就把他加入到列表的第一个元素，也就是访问成功。

{
IP1:[第三次请求时间，第二次请求时间,第一次请求时间,],
IP2:[第二次请求时间,第一次请求时间,],
.....
}

节流源码分析

首先请求进来走dispatch方法

def dispatch(self, request, *args, **kwargs):
    """
    `.dispatch()` is pretty much the same as Django's regular dispatch,
    but with extra hooks for startup, finalize, and exception handling.
    """
    self.args = args
    self.kwargs = kwargs
    # 请求模块,封装了Django原生的请求
    request = self.initialize_request(request, *args, **kwargs)
    self.request = request
    self.headers = self.default_response_headers  # deprecate?

    try:
        # 版本，认证，权限，限流，版本等功能都在initial里边
        self.initial(request, *args, **kwargs)

        # Get the appropriate handler method
        # 去http_method_names找请求的方法，如果没有找到，触发异常
        # http_method_names = ['get', 'post', 'put', 'patch', 'delete', 'head', 'options', 'trace']
        if request.method.lower() in self.http_method_names:
            handler = getattr(self, request.method.lower(),
                              self.http_method_not_allowed)
        else:
            handler = self.http_method_not_allowed

        response = handler(request, *args, **kwargs)

    except Exception as exc:
        # 异常处理模块
        response = self.handle_exception(exc)

    # 进行请求的渲染，为什么postman中测试返回字符串，
    # 浏览器中测试返回一个Django rest framework那个漂亮的页面呢，就是这里实现的
    self.response = self.finalize_response(request, response, *args, **kwargs)
    return self.response

频率控制在initial方法里边，点进去

def initial(self, request, *args, **kwargs):
    """
    Runs anything that needs to occur prior to calling the method handler.
    """
    self.format_kwarg = self.get_format_suffix(**kwargs)

    # Perform content negotiation and store the accepted info on the request
    neg = self.perform_content_negotiation(request)
    request.accepted_renderer, request.accepted_media_type = neg

    # Determine the API version, if versioning is in use.
    version, scheme = self.determine_version(request, *args, **kwargs)
    request.version, request.versioning_scheme = version, scheme

    # Ensure that the incoming request is permitted
    # 进行认证
    self.perform_authentication(request)
    # 进行限流
    self.check_permissions(request)
    # 节流
    self.check_throttles(request)

里边有个check_throttles方法，这个就是用来做节流的。

def check_throttles(self, request):
    """
    Check if request should be throttled.
    Raises an appropriate exception if the request is throttled.
    """
    throttle_durations = []
    """
    check_throttles里边有个get_throttles方法，
    """
    for throttle in self.get_throttles():
        if not throttle.allow_request(request, self):
            throttle_durations.append(throttle.wait())

    if throttle_durations:
        # Filter out `None` values which may happen in case of config / rate
        # changes, see #1438
        durations = [
            duration for duration in throttle_durations
            if duration is not None
        ]

        duration = max(durations, default=None)
        self.throttled(request, duration)

check_throttles里边有个get_throttles方法，它和认证，以及权限的源码类似，也是去实例化配置文件里边的类列表，并返回实例化后的对象列表，如果我们没有自己自定义情况的下，它就是去实例化配置文件里边的频率的类列表，并返回实例化后的节流对象列表，然后调用节流对象的allow_request方法，如果返回True，就可以访问，返回False就抛出异常。我们接下来一步步点进源码去看，发现他确实是这样子的。

def get_throttles(self):
    """
    Instantiates and returns the list of throttles that this view uses.
    """
    # get_throttles方法里面通过列表生成式，生成一个频率的类的列表
    return [throttle() for throttle in self.throttle_classes]

所以我们在我们的类里边配置上throttle_classes，那么就用我们自己的。

默认是空列表

'DEFAULT_THROTTLE_CLASSES': [],

频率控制的内置类

我们主要使用BaseThrottle，SimpleRateThrottle。

BaseThrottle源码分析：

class BaseThrottle:
    """
    Rate throttling of requests.
    """

    def allow_request(self, request, view):
        """
        Return `True` if the request should be allowed, `False` otherwise.
        """
        raise NotImplementedError('.allow_request() must be overridden')

    def get_ident(self, request):
        """
        Identify the machine making the request by parsing HTTP_X_FORWARDED_FOR
        if present and number of proxies is > 0. If not use all of
        HTTP_X_FORWARDED_FOR if it is available, if not use REMOTE_ADDR.
        """
        # 获取IP作为标识
        xff = request.META.get('HTTP_X_FORWARDED_FOR')
        remote_addr = request.META.get('REMOTE_ADDR')
        num_proxies = api_settings.NUM_PROXIES

        if num_proxies is not None:
            if num_proxies == 0 or xff is None:
                return remote_addr
            addrs = xff.split(',')
            client_addr = addrs[-min(num_proxies, len(addrs))]
            return client_addr.strip()

        return ''.join(xff.split()) if xff else remote_addr

    def wait(self):
        """
        Optionally, return a recommended number of seconds to wait before
        the next request.
        """
        # 返回秒数，表示需要等多少秒
        return None

基本使用
从源码中可以看到allow_request必须要重写，否则会报错。get_ident用于获取标识（IP），wait返回秒数，表示等多少秒

算法实现
import time
VISIT_RECORD = {}

class VisitThrottle(object):
	"""60s内只能访问3次"""

	def __init__(self):
		self.history = None

	def allow_request(self,request,view):
		# 1. 获取用户IP
		remote_addr = request.META.get('REMOTE_ADDR')
		ctime = time.time()
		if remote_addr not in VISIT_RECORD:
			VISIT_RECORD[remote_addr] = [ctime,]
			return True
		history = VISIT_RECORD.get(remote_addr)
		self.history = history

		while history and history[-1] < ctime - 60:
			history.pop()

		if len(history) < 3:
			history.insert(0,ctime)
			return True

		# return True    # 表示可以继续访问
		# return False # 表示访问频率太高，被限制

	def wait(self):
		"""
		还需要等多少秒才能访问
		:return:
		"""
		ctime = time.time()
		return 60 - (ctime - self.history[-1])

class AuthView(APIView):
	"""
	用于用户登录认证
	"""
	authentication_classes = []
	permission_classes = []
	throttle_classes = [VisitThrottle,]

	def post(self,request,*args,**kwargs):


		ret = {'code':1000,'msg':None}
		try:
			user = request._request.POST.get('username')
			pwd = request._request.POST.get('password')
			obj = models.UserInfo.objects.filter(username=user,password=pwd).first()
			if not obj:
				ret['code'] = 1001
				ret['msg'] = "用户名或密码错误"
			# 为登录用户创建token
			token = md5(user)
			# 存在就更新，不存在就创建
			models.UserToken.objects.update_or_create(user=obj,defaults={'token':token})
			ret['token'] = token
		except Exception as e:
			ret['code'] = 1002
			ret['msg'] = '请求异常'

		return JsonResponse(ret)

SimpleRateThrottle源码，SimpleRateThrottle帮我们把上面我们自己的逻辑写好了

class SimpleRateThrottle(BaseThrottle):
    """
    A simple cache implementation, that only requires `.get_cache_key()`
    to be overridden.

    The rate (requests / seconds) is set by a `rate` attribute on the View
    class.  The attribute is a string of the form 'number_of_requests/period'.

    Period should be one of: ('s', 'sec', 'm', 'min', 'h', 'hour', 'd', 'day')

    Previous request information used for throttling is stored in the cache.
    """
    cache = default_cache
    timer = time.time
    cache_format = 'throttle_%(scope)s_%(ident)s'
    scope = None
    THROTTLE_RATES = api_settings.DEFAULT_THROTTLE_RATES

    def __init__(self):
        if not getattr(self, 'rate', None):
            self.rate = self.get_rate()
        self.num_requests, self.duration = self.parse_rate(self.rate)

    def get_cache_key(self, request, view):
        """
        Should return a unique cache-key which can be used for throttling.
        Must be overridden.

        May return `None` if the request should not be throttled.
        """
        # 这个方法必须被重写，否则会报错
        raise NotImplementedError('.get_cache_key() must be overridden')

    def get_rate(self):
        """
        Determine the string representation of the allowed request rate.
        """
        if not getattr(self, 'scope', None):
            msg = ("You must set either `.scope` or `.rate` for '%s' throttle" %
                   self.__class__.__name__)
            raise ImproperlyConfigured(msg)

        try:
            return self.THROTTLE_RATES[self.scope]
        except KeyError:
            msg = "No default throttle rate set for '%s' scope" % self.scope
            raise ImproperlyConfigured(msg)

    def parse_rate(self, rate):
        """
        Given the request rate string, return a two tuple of:
        <allowed number of requests>, <period of time in seconds>
        """
        # 从配置文件中取到rate后，进行解析
        if rate is None:
            return (None, None)
        num, period = rate.split('/')
        num_requests = int(num)
        duration = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400}[period[0]]
        return (num_requests, duration)

    def allow_request(self, request, view):
        """
        Implement the check to see if the request should be throttled.

        On success calls `throttle_success`.
        On failure calls `throttle_failure`.
        """
        # 这是主要书写节流逻辑的地方
        if self.rate is None:
            return True

        # 获得key
        self.key = self.get_cache_key(request, view)
        if self.key is None:
            return True

        # 从访问列表里面获取，cache是Django内置的缓存
        self.history = self.cache.get(self.key, [])
        self.now = self.timer()

        # Drop any requests from the history which have now passed the
        # throttle duration
        while self.history and self.history[-1] <= self.now - self.duration:
            self.history.pop()
        if len(self.history) >= self.num_requests:
            return self.throttle_failure()
        return self.throttle_success()

    def throttle_success(self):
        """
        Inserts the current request's timestamp along with the key
        into the cache.
        """
        self.history.insert(0, self.now)
        self.cache.set(self.key, self.history, self.duration)
        return True

    def throttle_failure(self):
        """
        Called when a request to the API has failed due to throttling.
        """
        return False

    def wait(self):
        """
        Returns the recommended next request time in seconds.
        """
        if self.history:
            remaining_duration = self.duration - (self.now - self.history[-1])
        else:
            remaining_duration = self.duration

        available_requests = self.num_requests - len(self.history) + 1
        if available_requests <= 0:
            return None

        return remaining_duration / float(available_requests)

使用方法
在util目录下新建一个专门用来进行频率控制的Python文件，继承SimpleRateThrottle，get_cache_key方法必须被重写，要把字典的键赋值给scope变量，然后配置文件里面，去指定频率限制，因为

from rest_framework.throttling import BaseThrottle,SimpleRateThrottle
class VisitThrottle(SimpleRateThrottle):
	scope = "Luffy"

	def get_cache_key(self, request, view):
		return self.get_ident(request)


class UserThrottle(SimpleRateThrottle):
	scope = "LuffyUser"

	def get_cache_key(self, request, view):
		return request.user.username

REST_FRAMEWORK = {
	# 全局使用的认证类
	"DEFAULT_AUTHENTICATION_CLASSES":['api.utils.auth.FirstAuthtication','api.utils.auth.Authtication', ],
	# "DEFAULT_AUTHENTICATION_CLASSES":['api.utils.auth.FirstAuthtication', ],
	# "UNAUTHENTICATED_USER":lambda :"匿名用户"
	"UNAUTHENTICATED_USER":None, # 匿名，request.user = None
	"UNAUTHENTICATED_TOKEN":None,# 匿名，request.auth = None
	"DEFAULT_PERMISSION_CLASSES":['api.utils.permission.SVIPPermission'],
	"DEFAULT_THROTTLE_CLASSES":["api.utils.throttle.UserThrottle"],
	"DEFAULT_THROTTLE_RATES":{
		"Luffy":'3/m',
		"LuffyUser":'10/m',
	}
}