Article Directory
As someone who regularly write a blog, I sometimes in the micro-blog information sharing content on the blog, do not know when to begin, the domestic Internet more and more lose faith, all the manufacturers are trying to create a "just could not advance" Isolated islands to achieve the purpose of enhancing "user stickiness". Take Weibo as an example. The external links in Weibo will always be converted into short addresses and cannot be redirected through the built-in browser of Weibo. Even if you open the link by manually copying the link, you still need at least two steps to see the "true face of Lushan". Drawing / copying the bad habits there are simple book , take the time to do a third-party links Jumplines page, except do not want to add a a label on it, you still have to copy paste manually. Frankly speaking, I think that the domestic Internet is losing faith. It seems that e-commerce, logistics, takeaway, taxi, payment... one after another gradually penetrates into all aspects of our lives, becoming a veritable "Internet +", but in today's flood of information, It’s getting harder and harder to find really valuable information...Since the external links are destined to be blocked, then I reluctantly follow the trend and send "long screenshots". So, next I will share with you the realization of the webpage "long screenshots" "The common ideas, I hope it will be helpful to friends who have similar troubles or needs.
Through the browser
To achieve a long screenshot of a webpage, obviously you are dealing with the webpage, and who is dealing with the webpage the most? Naturally it is the browser we use every day! Fortunately, whether it is Chrome or Firefox, we can use them to realize this idea.
Chrome
For Chrome, we only need "F12" to open the developer tools, and find the "Console" tab in it, enter the Ctrl + Shift + Pcommand in the place where the JavaScript script is usually entered (that is, the Console tab) , and then you will get a similar VSCode The input window of the command line experience, next, type: Capture full size screenshotand press Enter. At this point, we can get a complete screenshot of the page. And if you want to intercept a portion of the page, you can specify a DOM element is selected in the same way enter the command: Capture node screenshot. In addition, more usually taken in the range of the visible content of the browser can be used: Capture screenshot. Perhaps compared to the general drag-and-drop screenshot tool, this solution is a bit clumsy and simple, but it can really implement our ideas perfectly, and there is no need to install any extensions or plug-ins.
Firefox
For Firefox, it has its own screenshot function and supports drag-and-drop screenshots. For those of us who need long screenshots, the only thing we need to do is to click a few clicks on the data. It is indeed simpler and friendly than typing the command line. One point, I personally prefer to use Firefox, because Chrome is changing from a dragon slayer to an evil dragon. In order to make Chrome not the only browser kernel in the world, I decided to support Firefox. In 2020, due to the epidemic, Mozila It laid off 25% of its workforce and about 250 people. This company, which almost relies on idealism to maintain the Gecko kernel, may no longer be able to compete with Google’s Chrome. We have all experienced the era when there was only one browser in the world. Its name It's called IE6, and it's sighing. It's simply the history of the romantic demise of the open Web.
Through Selenium
In my cognition, where there is a browser, there is a crawler, and where there is a crawler, there is Selenium. How can the originally good UI automation testing framework help Chun to become a crawler for abuse? In fact, the main reason is that it provides an environment that can interact with the browser. In a sense, Selenium , PhantomJS, and Playwright can all be considered similar technologies. Here we take Selenium as an example, and use Selenium to achieve web page length There are two main ways to take screenshots: one is to construct a large enough browser, and then call the save_screenshot()method to take the screenshot; the other is to scroll the screenshot through the "drag and drop" scroll bar, and then PILto stitch together. Look at the specific code implementation:
def save_screenshot(url, fp_pic):
fireFoxOptions = webdriver.FirefoxOptions()
fireFoxOptions.set_headless()
driver = webdriver.Firefox(firefox_options=fireFoxOptions)
driver.get(url)
time.sleep(1)
# 设置浏览器宽度和高度
width = driver.execute_script(
"return document.documentElement.scrollWidth"
)
height = driver.execute_script(
"return document.documentElement.scrollHeight"
)
driver.set_window_size(width, height)
time.sleep(1)
# 截图
driver.save_screenshot(fp_pic)
driver.close()
Here I am using the Firefox driver. If you like Chrome, you can do it according to personal preference. Here I assume that you have mastered Python and Selenium. If you need a little auxiliary knowledge, you can refer to this blogger’s article: As a technical house, This is how you chase the ghost off the blade . This method of "long screenshot" is very simple to implement, but because it needs to construct a very "large" browser, if the page adaptation is not done well, the problem of page element deformation may occur. Secondly, this The size of the pictures generated by this method is generally relatively large, so the overall view is mainly due to these two shortcomings. The implementation of "scrolling screenshots" is a little more complicated, because it involves a small part of calculations:
def save_screenshot2(url, fp_pic):
fireFoxOptions = webdriver.FirefoxOptions()
fireFoxOptions.set_headless()
driver = webdriver.Firefox(firefox_options=fireFoxOptions)
driver.fullscreen_window() # 全屏窗口
driver.get(url)
window_height = driver.get_window_size()['height'] # 窗口高度
page_height = driver.execute_script(
'return document.documentElement.scrollHeight'
) # 页面高度
driver.save_screenshot('temp.png')
if page_height > window_height:
n = page_height // window_height # 需要滚动的次数
base_mat = np.atleast_2d(Image.open('temp.png')) # 打开截图并转为二维矩阵
for i in range(n):
driver.execute_script(
f'document.documentElement.scrollTop={window_height * (i+1)};'
)
time.sleep(.5)
driver.save_screenshot(f'temp_{i}.png') # 保存截图
mat = np.atleast_2d(Image.open(f'temp_{i}.png')) # 打开截图并转为二维矩阵
base_mat = np.append(base_mat, mat, axis=0) # 拼接图片的二维矩阵
Image.fromarray(base_mat).save(fp_pic, format='PNG')
os.remove(f'temp_{i}.png')
os.remove('temp.png')
driver.quit()
There is no big problem with this scheme, but if your webpage has a design with a fixed head when the page scrolls, such as a style similar to a blogger’s blog, this scheme will have a little problem at this time, every time you intercept The head part will be included, which is a little different from the effect we want to achieve. If the height of the head can be calculated, and this height is taken into account when taking screenshots or stitching, this problem can be completely solved, but this problem It changed from a general problem to a partial problem. Sure enough, there is no perfect solution...
Through JavaScript
Someone might say, the blogger is eccentric, why has Python come out, and JavaScript, one of the three front-end swordsmen, has not yet appeared? Well, I want to say about this-you don't need to say, I know it is **"Life is short, I use Python"**? There is a library called html2canvas in the front-end world of others. Have you heard of a blogger? I smiled. I didn't look at my Rolex because I don't have a Rolex. Well, since this library is mentioned here, let's talk about the realization ideas of this library. Didn't people say it? Everything that can be achieved can finally be achieved with JavaScript. Let's take a look at the specific code implementation. Here, first prepare an HTML file:
<!DOCTYPE html>
<head>
<script src='./html2canvas.min.js'></script>
<script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js"></script>
</head>
<body style="overflow: hidden;">
<div id="app" style="height: 768px; overflow: hidden;">
请输入URL: <input type="text" v-model="url">
<button v-on:click="capture">截取</button>
<hr>
<iframe
id="view"
v-bind:src='url'
width="100%"
height="100%"
frameborder="0"
ref="view">
</iframe>
</div>
</body>
It's very simple. Enter an address, preview it through an iframe, and click the button to take a screenshot. The JavaScript code is given below:
<script>
var vm = new Vue({
el: '#app',
data: {
url: 'https://regularly-archive.github.io/2020/Matrix/',
},
methods: {
capture: function() {
var self = this;
var iframe = self.$refs.view.contentWindow;
var iframeBody = iframe.document.getElementsByTagName('body')[0]
html2canvas(iframeBody).then(canvas => {
document.body.appendChild(canvas);
//canvas转图片
let canvasImg = canvas.toDataURL("image/png");
//模拟下载
var a = document.createElement('a')
a.href = canvasImg;
a.download = self.url;
let event = document.createEvent("MouseEvents")
event.initMouseEvent("click", true, false,
window, 0, 0, 0, 0, 0, false, false, false, false, 0, null)
a.dispatchEvent(event)
});
}
}
});
</script>
Results are as follows, you can click here to access the online demo DEMO:
The use of iframes here may introduce cross-domain problems. You can refer to this article of mine: Let’s talk about cross-domain love and hatred in the front-end , and html2canvas itself provides solutions to cross-domain problems. You can refer to here: http://html2canvas.hertzen.com/configuration .
Through third-party tools
I know that programmers like to toss on their own. If it is something that has never been seen before, I suggest to toss, because dreams are still necessary, what if it is realized? And our circle also has a classic saying, "Don’t make wheels repeatedly". Therefore, the blogger found a few wheels for your reference. Those who don’t like to write code in winter can collect these tools. , This winter is too cold, how cold is it? I probably don’t want to laugh when I hear a joke. In the words of Teacher Luo Xiang, this is called a funny attempt.
wkhtml2image
The wkhtml series , a command line tool, can convert local HTML files or web pages pointed to by remote URLs into images. There is also wkhtml2pdf in this series of products. As the name suggests, web pages can be converted to PDF. There is basically no problem in actual use. The output image 1: 1. The only disadvantage of restoring webpages is that they occasionally lose styles, especially when third-party JavaScript or CSS is introduced into the page. The overall remote URL is a bit more stable than the local HTML. The recommendation factor is 4 stars.
PickFrom
PickFrom , an online web page transfer service, just fill in the URL and click the button. It provides the function of previewing part of the picture for free. You need to pay a certain fee for viewing and downloading the complete picture. The service quality is OK, but it is not suitable for us. Children from poor families who are forced to "white prostitutes", local tyrants are free, recommendation coefficient: 4 stars
Tiomg
Tiomg , next is the main recommendation of blog. It provides the same service as PickFrom . The only difference is that it is completely free. I now mainly use this tool to generate "long screenshots". Yes, I betrayed the above. The code I wrote, why do you need to reinvent the wheel? Sometimes I don’t understand why domestic companies like "big and complete" software, and they want to have all the features of their competitors, but it is clear that everyone has "rolled" into this, why not try a differentiated route What? Maybe it's because there is too much low-end competition! Recommendation factor: 5 stars
There is really no motivation to write technical things in winter! Regarding the topic of "long screenshots", I have been following and experimenting almost a week ago, so please allow me to write such a "hydrology" occasionally next time! Regarding "Video is a series that can't be P", because Dlib is really annoying to install, and the 68 feature point algorithm provided by OpenCV currently only supports C++, it will inevitably take a while to research. Okay, let's write this blog here for now. , The blogger goes to hibernation first, bye!