使用DJANGO框架实现
1、日志路径的录入
首先用户需要录入站点的名称，以及日志的网络路径，作为对应关系用于将来统计
models.py加入
class systemloginfo(models.Model):
id = models.IntegerField(primary_key=True)
dnsname = models.CharField(max_length=50)
logsurl = models.CharField(max_length=200)
urls.py加入
url(r'^addlogs/$',views.addlogs,name='addlogs'),
views.py加入
def addlogs(request):
if request.method=="POST":
dnss=request.POST.get("dnss")
logsurl=request.POST.get("logsurl")
try:
addlistlogs=systemloginfo(dnsname=dnss,logsurl=logsurl)
addlistlogs.save()
return render(request,"addlogs.html",{"login_err":"okay"})
except Exception:
return render(request,"addlogs.html",{"login_err":"fail"})
else:
return render(request,"addlogs.html",{"login_err":"noset"})
templates目录下建立前端的HTML文件addlogs.html，如下：
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>录入应用日志的URL</title>
</head>
<body>
<form id="addlogsform" action="{% url 'addlogs' %}" method="POST"> {% csrf_token %}
应用域名<input type="text" class="form-control" name="dnss" placeholder="应用域名"></br>
日志路径<input type="text" class="form-control" name="logsurl" placeholder="日志路径"></br>
</br>
<button class="btn btn-success btn-block" type="submit">
<b>提交</b>
</button>
<h4 style="color: red"><b>{{ login_err }}</b></h4>
</form>
</body>
</html>
因为统计访问量采用了PYTHON内置的open方法打开日志统计，在录入应用URL的时候须遵循以下格式，如下：
如果一个应用部署在了多台服务器做负载均衡的话，可以分别录入各个的日志文件的路径
日志路径的录入模块完成
2、访问量统计的实现，暂时分为按月度统计和按季度统计
urls.py加入
url(r'^selectlogs/$',views.selectlogs,name='selectlogs'),
views.py加入
def selectlogs(request):
try:
dnslists=models.systemloginfo.objects.all().values('dnsname').distinct()
except Exception:
return render(request,"selectlogs.html",{"login_err":"LOAD DATA FAIL"})
if request.method=="POST":
counts=0
if request.POST.has_key('yd'):
try:
dnss=request.POST.get("dnsname")
lists=models.systemloginfo.objects.all().filter(dnsname=dnss).values('dnsname','logsurl')
counts=wbemtest.recounts(lists,30)
str30="THE MOMTH TOTAL CLICK "+dnss+" "+str(counts)
except Exception:
return render(request,"selectlogs.html",{"login_err":dnss,"lists":dnslists})
return render(request,"selectlogs.html",{"login_err":str30,"lists":dnslists})
if request.POST.has_key('jd'):
try:
dnss=request.POST.get("dnsname")
lists=models.systemloginfo.objects.all().filter(dnsname=dnss).values('dnsname','logsurl')
counts=wbemtest.recounts(lists,90)
str90="THE QUARTER TOTAL CLICK "+dnss+" "+str(counts)
except Exception:
return render(request,"selectlogs.html",{"login_err":"fail90","lists":dnslists})
return render(request,"selectlogs.html",{"login_err":str90,"lists":dnslists})
if request.POST.has_key("alljd"):
try:
listlog=wbemtest.reallcounts(dnslists,90)
except Exception:
return render(request,"selectlogs.html",{"login_err":"failall90","lists":dnslists})
return render(request,"selectlogs.html",{"jdlists":listlog,"lists":dnslists})
if request.POST.has_key("allyd"):
try:
listlog=wbemtest.reallcounts(dnslists,30)
except Exception:
return render(request,"selectlogs.html",{"login_err":"failall30","lists":dnslists})
return render(request,"selectlogs.html",{"ydlists":listlog,"lists":dnslists})
else:
try:
return render(request,"selectlogs.html",{"lists":dnslists})
except Exception:
return render(request,"selectlogs.html",{"login_err":"LOAD DATA FAIL"})
views.py当中调用的wbemtest.recounts以及wbemtest.reallcounts方法实现如下：
def recounts(lists,days):
a=0
for row in lists:
a1=0
logsurl=row["logsurl"]
try:
a1=dj27.many.realls(logsurl,days)
a=a1+a
except Exception:
return 0
return a
def reallcounts(lists,days):
loglists=[]
try:
for row in lists:
rowdict={}
dnsname=row["dnsname"]
urllist=dj27.models.systemloginfo.objects.all().filter(dnsname=dnsname).values('dnsname','logsurl')
nums=recounts(urllist,days) #此处再次调用了上面的recounts方法
rowdict["dnsname"]=dnsname
rowdict["nums"]=nums
loglists.append(rowdict)
return loglists
except Exception:
return loglists
统计IIS站点访问量实际上就是打开日志文件除去#标记，一行一行地数，由于日志文件众多，上面调用的dj27.many.realls方法使用了多线程设计，加快计算的速度，many.py文件如下：
# -*- coding: utf-8 -*-
import threading
import time
import datetime
import os,sys
reload(sys)
class MyThread(threading.Thread):
def __init__(self,func,args=()):
super(MyThread,self).__init__()
self.func = func
self.args = args
def run(self):
self.result = self.func(*self.args)
def get_result(self):
try:
return self.result
except Exception:
return None
def redallfiles(dirs):
return os.listdir(dirs)
def isfiles(files,days):
lastmodifytime = os.stat(files).st_mtime
endfiletime = time.time() - 3600 * 24 * days
if endfiletime > lastmodifytime:
return False
else:
return True
def countsfile(files,days):
a=0
if isfiles(files,days):
readfiles=open(files,"r")
for line in readfiles:
if line.startswith("#"):
continue
else:
a=a+1
readfiles.close()
else:
pass
return a
def counts(logsurl,days):
a=0
files=redallfiles(logsurl)
for f in files:
if isfiles(logsurl+f,days):
readfiles=open(logsurl+f,'r')
for line in readfiles:
if line.startswith("#"):
continue
else:
a=a+1
readfiles.close()
else:
pass
return a
def recounts(lists,days):
a=0
for row in lists:
a1=0
logsurl=row["logsurl"]
try:
a1=counts(logsurl,days)
a=a1+a
except Exception:
pass
return a
def realls(urls,days):
threads = []
filelists=redallfiles(urls)
files = range(len(filelists))
for i in files:
everyfile=urls+filelists[i]
# t = threading.Thread(target=countsfile,args=(everyfile,180,))
t = MyThread(countsfile,args=(everyfile,days,))
threads.append(t)
t.start()
alls=0
for t in threads:
t.join()
a1=t.get_result()
alls=a1+alls
return alls
templates目录下建立前端的HTML文件selectlogs.html，如下：
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>统计应用访问数量</title>
</head>
<body>
<form id="selectlogsform" action="{% url 'selectlogs' %}" method="POST"> {% csrf_token %}
<select name="dnsname">
{% for systems in lists %}
<option value ={{ systems.dnsname }}>{{ systems.dnsname }}</option>
{% endfor %}
</select></br>
</br>
<button class="btn btn-success btn-block" type="submit" name="yd">
<b>查询月度访问量</b>
</button>
</br></br>
<button class="btn btn-success btn-block" type="submit" name="jd">
<b>查询季度访问量</b>
</button>
</br>
</br>
</br></br>
<button class="btn btn-success btn-block" type="submit" name="allyd">
<b>查询全部应用月度访问量</b>
</button>
</br></br>
<button class="btn btn-success btn-block" type="submit" name="alljd">
<b>查询全部应用季度访问量</b>
</button>
</br>
</br>
{% for yddict1 in ydlists %}
<h4 style="color: black"><b>应用名称:{{ yddict1.dnsname }}</b> 月度访问数量：{{ yddict1.nums }}</h4>
<hr>
{% endfor %}
{% for jddict1 in jdlists %}
<h4 style="color: black"><b>应用名称:{{ jddict1.dnsname }}</b> 季度访问数量：{{ jddict1.nums }}</h4>
<hr>
{% endfor %}
{{ login_err }}
</form>
</body>
</html>
点击按钮查询对应的访问量：
也可以一次统计全部应用的访问量
