Use Squid to build an intranet video caching system

videocache address: http://cachevideos.com/documentation/introduction

Software requirements: videocache, squid-2.7, apache2, redis, python2.7

Operating System: Ubuntu

File cache path: / var / www/html/videocache/

Steps:

==================

1. Install Python: Ubuntu system comes with Python 2.7 version, skip this step

2. Install Mysql (this step is not used, you can skip it directly)

$ apt-get install mysql-server mysql-client libmysqlclient-dev

$ mysql -uroot -proot

$ create database videocache

3. Install apache2. After the installation is complete, apache2 will start automatically. You can visit http://ip/ to test whether apache2 is successfully installed.

$ apt-get install apache2

4. Install squid2.7

$ wget http://www.squid-cache.org/Versions/v2/2.7/squid-2.7.STABLE9.tar.gz

$ tar -xvf squid-2.7.STABLE9.tar.gz

$ ./configure --prefix=/usr/local/squid --sysconfdir=/usr/local/squid/etc --bindir=/usr/local/squid/bin --sbindir=/usr/local/squid/sbin --mandir=/usr/local/squid/share/man --enable-gnuregex --enable-carp --enable-async-io=80 --enable-removal-policies=heap,lru --enable-icmp --enable-delay-pools --enable-useragent-log --enable-referer-log --enable-kill-parent-hack --enable-snmp --enable-arp-acl --enable-htcp --enable-cache-digests --enable-default-err-language=Simplify_Chinese --enable-err-languages="Simplify_Chinese" --enable-poll --enable-linux-netfilter --disable-ident-lookups --enable-underscores --enable-auth="basic" --enable-basic-auth-helpers="NCSA" --enable-external-acl-helpers="ip_user" --enable-x-accelerator-vary

$ make

$ make install

5. Modify the configuration squid file and replace it with the following

# --BEGIN-- videocache config for squid
#
# PASTE THIS FILE AT THE TOP OF YOUR SQUID CONFIGURATION FILE
#
# (C) Copyright Kulbir Saini < [email protected] >
# Product Website : http://cachevideos.com/
#
# Videocache Version : 3.0
#
# This configuration is for Squid versions 2.x
url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 15
url_rewrite_concurrency 15

acl vc_deny_url url_regex -i \.blip\.tv\/(.*)filename \.hardsextube\.com\/videothumbs \.xtube\.com\/(.*)(Thumb|videowall)
acl vc_deny_url url_regex -i \.(jpg|jpeg|gif|png|tiff|mp3|exif|riff|bmp|webp)\?

acl vc_url url_regex -i \/youku\/[0-9A-Z]+\/[0-9A-Z\-]+\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg)
acl vc_url url_regex -i \/(.*)key=[a-z0-9]+(.*)\.flv
acl vc_url url_regex -i \-xh\.clients\.cdn[0-9a-zA-Z]?[0-9a-zA-Z]?[0-9a-zA-Z]?\.com\/data\/(.*)\.flv
acl vc_url url_regex -i \.android\.clients\.google\.com\/market\/GetBinary\/
acl vc_url url_regex -i cs(.*)\.vk\.me\/(.*)/([a-zA-Z0-9.]+)\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg)
acl vc_url url_regex -i video(.*)\.rutube\.ru\/(.*)/([a-zA-Z0-9.]+)\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg)Seg[0-9]+-Frag[0-9]+

acl vc_dom_r dstdom_regex -i msn\..*\.(com|net)
acl vc_dom_r dstdom_regex -i msnbc\..*\.(com|net)
acl vc_dom_r dstdom_regex -i video\..*\.fbcdn\.net
acl vc_dom_r dstdom_regex -i myspacecdn\..*\.footprint\.net

acl vc_dom dstdomain .stream.aol.com .5min.com .msn.com .blip.tv .dmcdn.net .break.com .vimeo.com .vimeocdn.com video.thestaticvube.com
acl vc_dom dstdomain .dailymotion.com .c.wrzuta.pl .v.imwx.com .mccont.com .myspacecdn.com video-http.media-imdb.com fcache.veoh.com
acl vc_dom dstdomain .hardsextube.com .public.extremetube.phncdn.com .redtubefiles.com .video.pornhub.phncdn.com .videos.videobash.com
acl vc_dom dstdomain .public.keezmovies.com .public.keezmovies.phncdn.com .slutload-media.com .public.spankwire.com .xtube.com
acl vc_dom dstdomain .public.youporn.phncdn.com .xvideos.com .tube8.com .public.spankwire.phncdn.com .pornhub.com

refresh_pattern \.video\.msn\.com\/.*\/entityid\/[a-zA-Z0-9_\-]+\? 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern \.android\.clients\.google\.com\/market\/GetBinary\/ 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern stream\.aol\.com\/(.*)/[a-zA-Z0-9]+\/(.*)\.(flv|mp4) 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern videos\.5min\.com\/(.*)/[0-9_]+\.(mp4|flv) 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern \.blip\.tv\/(.*)\.(m4v|mp4|flv) 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern proxy[a-z0-9\-]?[a-z0-9]?[a-z0-9]?[a-z0-9]?\.dailymotion\.com\/(.*)\.(flv|on2|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern vid\.akm\.dailymotion\.com\/(.*)\.(flv|on2|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern \.dmcdn\.net\/(.*)\.(flv|on2|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern video\.(.*)\.fbcdn\.net\/(.*)/[0-9_]+\.(mp4|flv|avi|mkv|m4v|mov|wmv|3gp|mpg|mpeg) 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern (.*)\.myspacecdn\.com\/(.*)\/[a-zA-Z0-9]+\/vid\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern (.*)\.myspacecdn\.(.*)\.footprint\.net\/(.*)\/[a-zA-Z0-9]+\/vid\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern c\.wrzuta\.pl\/w[a-zA-Z0-9]+\/[a-zA-Z0-9]+$ 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern \.hardsextube\.com\/.*\/.*\.(flv|mp4|avi|mkv|mp3|rm|rmvb|m4v|mov|wmv|3gp|mpg|mpeg) 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern -xh\.clients\.cdn[0-9a-zA-Z]?[0-9a-zA-Z]?[0-9a-zA-Z]?\.com\/data\/(.*)\.flv 20 80% 40 ignore-no-cache override-expire override-lastmod ignore-private

acl vc_deny_url url_regex -i crossdomain.xml
acl vc_method method GET
acl vc_header req_header X-Requested-With -i videocache
url_rewrite_access deny !vc_method
url_rewrite_access deny vc_header
url_rewrite_access deny vc_deny_url
url_rewrite_access allow vc_dom
url_rewrite_access allow vc_url
url_rewrite_access allow vc_dom_r
redirector_bypass on
#These lines are needed.
strip_query_terms off
maximum_object_size 200 MB
acl this_machine src 127.0.0.1
http_access allow this_machine
# --END-- videocache config for squid

    acl localnet src 192.0.0.0/8
    acl all src all
    acl manager proto cache_object
    acl localhost src 127.0.0.1/32
    acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
    acl SSL_ports port 443
    acl CONNECT method CONNECT

    http_access allow manager localhost
    http_access deny manager

    http_access deny CONNECT !SSL_ports
    http_access allow localnet
    http_access deny all

    http_port 3128
    hierarchy_stoplist cgi-bin ?
    access_log /usr/local/squid/var/logs/access.log squid
    refresh_pattern ^ftp: 1440 20% 10080
    refresh_pattern ^gopher: 1440 0% 1440
    refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
    refresh_pattern . 0 20% 4320
    acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
    upgrade_http0.9 deny shoutcast
    acl apache rep_header Server ^Apache
    broken_vary_encoding allow apache
    coredump_dir /usr/local/squid/var/cache

    cache_mem 400 MB
    maximum_object_size 4096000 KB
    minimum_object_size 0 KB
    maximum_object_size_in_memory 2560 KB

    cache_dir ufs /usr/local/squid/var/cache-data 20480 16 256
    cache_swap_low 80
    cache_swap_high 97
    strip_query_terms off
    request_body_max_size 5 MB
    memory_pools on
    memory_pools_limit 50 MB

6. Configure the local DNS server (this step can be omitted), add namesever to point to the local DNS

$ vi /etc/resolve.conf

nameserver 114.114.114.114

nameserver 8.8.8.8

search localdomain

7. Install redis and start automatically after the installation is complete

$ apt-get install redis-server

8. Install videocache, videocache depends on python and requires many libraries

$ tar -xvf videocache-master.tar

$ cd videocache-master

$ ./install.sh

If the IP address of the proxy server changes, please modify cache_host=your IP address in the /etc/videocache.conf file

During the installation process, the missing python class library will be installed automatically. For example, the missing class library name is A. You can also manually execute apt-get install python-A to install it.

After the installation is complete, five will pop up, just do it

$ vc-update

$ apachectl -k restart

$ vc-scheduler -s restart

8. Start squid

$ cd /usr/local/squid/

mod chmod 777 ./var

$ ./sbin/squid -k shutdown

$ ./sbin/squid -z

$ ./sbin/squid -d 1

The first time you run it, run squid -z to create a cache folder. The cache folder is created under /usr/local/squid/var, so you need to run chmod 777 /usr/local/squid/var to give the folder permission before creating the cache.

Then run squid -d 1 to open squid.

If you want to shut down squid, run squid -k shutdown is a safe shutdown.

$ /usr/local/squid/sbin/squid

Error at startup, as follows

FATAL: Cannot open '/usr/local/squid/var/logs/access.log' for writing.

The parent directory must be writeable by the

user 'nobody', which is the cache_effective_user

set in squid.conf.

Squid Cache (Version 2.7.STABLE9): Terminated abnormally.

CPU Usage: 0.005 seconds = 0.001 user + 0.004 sys

Maximum Resident Size: 15008 KB

Page faults with physical i/o: 0

Modify permissions to solve:

$ chown -R nobody /usr/local/squid/var/logs/access.log

$echo>var/logs/store.log

$ chown -R nobody /usr/local/squid/var/logs/store.log

$ ./sbin/squid -z

$ ./sbin/squid -d 1

==================

Now that the caching system has been activated, all cached files are cached in the /var/www/html/videocache directory, and the cached content can be viewed through http://ip/videocache/

The default port for the cache server is 3128

===========================

The following are commonly used commands for squid, squidclient is in the bin directory of squid installation

squidclient - p 80 mgr : info       #Get squid running status information: 
squidclient - p 80 mgr : mem      #Get squid memory usage: 
squidclient - p 80 mgr : objects #Get the cached list of squid, be careful when using it, it may Cause squid to crash : 
squidclient -p 80 mgr : diskd   #Get squid 's disk   usage : 
squidclient -p 80 -m PURGE http://www.361way.com#Force update a url, that is, clean up the squid of a connection

Use Squid to build an intranet video caching system

Guess you like