Article directory
[Author homepage]:Wu Qiulin
[Author introduction]: A high-quality creator in the Python field, an expert on Alibaba Cloud Blog, and an expert on Huawei Cloud Sharing. Long-term commitment to research and development in the field of Python and crawlers!
[Recommended by the author]: Friends who are interested in JS reversal can pay attention to "Reptile JS Reverse Practice" , for distribution Friends who are interested in the new crawler platform can follow "Practical Construction and Development of Distributed Crawler Platform"
There will also be verification code penetration and APP that will be continuously updated in the future. A series of articles on reverse engineering and Python fields
1. Write in front
A long time ago we did a project to collect Android APK resources shared in forums. You can see from the picture belowRemoving advertising, cracking, internal supply, VIP... Various magically modified APP versions emerge in endlessly in the forum. If we talk about security Thinking from this perspective, these APK resources are of great research value!
These resources are mainly shared and disseminated in the forum through third-party storage media (network disk, cloud disk), and the link address will be embedded in the post content in an unordered manner! To extract real and valid link addresses and passwords from text content, the method used in the early days was based on rules, a rule base of regular + features to hit and identify!
There was no GPT at that time, but now it is much easier to deal with such a scenario! ! !
This issue mainly talks about obtaining the download link of Lanzuo Cloud resources (requires password, multi-resource sharing, single resource sharing) and more Link extraction analysis for each situation
The download and extraction methods for Baidu Netdisk and Tianyi Cloud Disk will also be updated later when there is time.
Fortunately, when I wrote this article, all the interfaces and parameters involved have not changed much compared to before! Because I wrote it with reference to the previous code~~
Before starting to read this article, please take a look at the pitfalls you have encountered before, in case you happen to encounter them as well, as shown below:
1. Since the APK package is stored in the cloud disk, the existence of the direct link is time-sensitive (<10 minutes)
2. In the verification step of network detection, Sign may appear Parameter verification error, please wait for a few seconds and try again several times to pass parameter verification
2. Single file extraction
This type of APK resource can be directly downloaded by clicking on the sharing link to jump to it. It is the same as the interface with password extraction, and the parameters are different, as shown below:
The sign parameter can be obtained in the response HTML code, and other parameters are fixed parameters, so there is basically no major difficulty. The request parameters are as follows:sign a>
What needs to be mentioned here is that in the JSON data of the interface request response, two fields, dom and url, need to be extracted and spliced together next time The requested link, the response data is as follows:
{“zt”:1,“dom”:“https://developer-oss.lanzouc.com”,“url”:“?UjRWaF1sADEACQszUWQCbltkU2sCuAaBV/hSvFOMU+8A5gXbAfdV5wXcU4pU5wKNBtcB4lefBKZQvgOcUZ0G4VL+VqBd5gDBAOYLc1FkAnNbalNwAjsGbVc8UjFTUFM7AGQFbwFtVTIFalM8VDUCNQZrATNXLgRgUCQDOFE/BjVSY1Y3XTUAYABgC2hRIwJzW3FTawJvBjRXYVJtUyBTYwAxBX0BbVUwBXxTZ1QwAmUGZQEzVz0EMlBuA2dRaAZjUmZWN11mADcAZgs5UWMCZ1s5U24CPAYwV2dSYFNuU2oAMAViAT5VMQU2UytUYQJzBjcBIld9BHVQMgN3UWQGY1JvVjBdMABnAGMLaFEwAjdbJ1MiAjQGaVc1UjJTMlNjADcFYgFuVTIFZVM3VDICMwZqASpXJgQgUDEDaVF6BjpSY1Y2XTAAZgBlC2tRNAIyWzNTYwJ7BnFXIFIjUzJTYwA1BWUBb1U1BWtTNFQ3AjIGawEiV30Eb1AnAzhRPAY3UmRWL10zAGwAeAtsUTcCMlsvU2cCZAYw”,“inf”:0}
The splicing method requires adding a string file:os.path.join(dom, ‘file’, url)
After getting the link for the next request, initiate a request, and a verification test will appear. You need to click the trigger here to continue the subsequent download operation, as shown below:
Analyze the packet capture of the verification and download operation in the above picture. You can see that it is also an Ajax request, and the sign parameter is also responding. The file parameter can also be obtained in the HTML code, as shown below:
After obtaining the parameters required for the above verification and detection request, submit the verification and download request to pass the verification, as shown below:
After passing the verification, the interface response data contains the download link of the final APK resource, which can be extracted directly. The response data is as follows:
{“ZT”: 1, “DOM”: NULL, “URL”: “https://c1026.LANOSSO.com/DF137b170d87e8a5968ef81693144f7c/65780215/2019/11/11/B0 apk? fn =%E5%85%A8 %E8%83%BD%E6%89%AB%E6%8F%8F%E7%8E%8B%E6%97%A0%E9%99%90%E6%AC%A1%E6%95%B0.apk ”}
There is another kind of page, that is, after clicking the sharing link of the cloud disk, it will jump directly to the following page:
In this case, the page type is just different. At this step, directly splice the above link containing file and request directly, and you will come to The step of verifying the detection page
3. Multiple file sharing and extraction
Another option is to share a link that contains multiple APK resource files. They are all different and you may need to download them all, as shown below:
First of all, it has an interface for loading more files. Let’s capture the packet and analyze it, as follows:
These parameters are also available in the response HTML code, as shown below:
As of now, you will find that Lanzuoyun’s parameters do not use difficult or complex encryption algorithms, nor do they have relatively strict risk control! The only tedious thing is that there are a lot of processes. The code for extracting multiple APK resource files is as follows, for reference only:
def get_first_match(self, pattern, text):
match = findall(pattern, text)
return match[0] if match else None
def get_more_apk(self, url, html, data: dict = {
}):
lx = self.get_first_match("'lx':(\\d{1})", html)
fid = self.get_first_match("'fid':(\\d{7})", html)
uid = self.get_first_match("'uid':'\\d{5}'", html)
pgs = self.get_first_match("pgs =(\\d{1})", html)
_k = self.get_first_match("'k':([0-9a-zA-Z]{6}),", html)
_t = self.get_first_match("'t':([0-9a-zA-Z]{6}),", html)
if _k and _t:
k = self.get_first_match(
"var {} = '(.*)';".format(_k), html)
t = self.get_first_match(
"var {} = '(.*)';".format(_t), html)
data = {
"lx": lx,
"fid": fid,
"uid": str(uid),
"pg": pgs,
"rep": '0',
"t": t,
"k": k,
"up": 1,
"vip": '0',
"webfoldersign": ""
}
if data:
moreapk_res = requests.post(
'https://pan.lanzoub.com/filemoreajax.php',
json=data,
headers=self.headers
).json()
_apk_json = moreapk_res.get('text', '')
if isinstance(_apk_json, list):
for _json in _apk_json:
apk_info = {
}
icon = _json.get('icon', '')
if icon == 'apk':
apk_info['url'] = urljoin(
self.host_url, _json['id'])
apk_info['title'] = _json['name_all']
apk_info['size'] = _json['size']
print(apk_info)
4. Password login extraction
The last one is a common resource that requires an extraction code (password). It is the same as the single file sharing process above, except that there is a p parameter, this parameter is the password, as shown below:
The sign parameter can also be obtained in the affected HTML code, as shown below:
After submitting the request carrying the password parameter above, you will enter the file sharing page, and then repeat the above process, first constructing the file carrying The link is requested, and then tested, and the final interface response data is as follows:
{“zt”:1,“dom”:“https://developer-oss.lanzouc.com”,“url”:“?BGJXaQ08Dz4BCFBoBzIHa1plUmpS6QC6CqBTsAfWUd4I71a/CdoHugn8UJlW6ga3Bu1UDlU/U2kHMwc3AW1bdARlV3ANOw99ATFQbwc/B2NaX1I+UmkAPwo/U2IHaVFpCDxWNAljB2QJeVBgVnEGOgZtVGBVOlNpBzgHNwFqW20EIldwDSAPZgFlUDYHYQc2Wi9SZ1I1AC0KPVNnB3RRYwhtVmUJMQc2CWZQZVZgBmQGaVQ3VW5TNQdmBzYBPFtoBGVXOQ1oDz0BZlAxB2YHN1o0UjVSPQA0Cm9TZgdsUX4IaVZxCT8HdQkqUHVWZwZ1BjVUNVU3U2AHNQc3AWpbbgQ9VzMNdg8vAT5Qawc2B2BaPVJmUjoAMgo/U2YHblFiCD9WNwliB30JcVAgVmQGawYrVGxVO1NmBzAHMAFuW2oEN1czDWEPYgFxUHMHIwdxWj1SZlI4ADsKO1NmB29RaAg8VjQJYQd1CSpQb1ZyBjoGbVRhVTxTfwczBzkBc1trBDZXMQ1+D2oBblAy”,“inf”:“\u4e2d\u534e\u7f8e\u98df\u8c31_58360.apk”}
5. Anti-hotlink processing
During the request process, you may encounter possible anti-leeching, mainly to protect resources and control traffic. Please handle it a little. The code reference is as follows:
iframe_element = response.xpath("//div[@class='iframe']")
# 防盗链处理
if 'plugin.php' in response.url or iframe_element:
share_link = response.xpath("//div[@class='iframe']//iframe/@src").extract_first()
# 拿到真实链接地址重新请求
yield scrapy.Request(
share_link,
callback=self.parse_handle,
meta=response.meta
)
6. Download and share resources
Here we use the open source warehouse YouTube-dl to download APK resources. The code reference is as follows:
def rename_hook(d):
if d['status'] == 'finished':
print(
'====> APK Download Fnish. {}'.format(
d.get(
'filename',
'')))
def download(apk_link, apk_name):
try:
ydl_opts = {
'outtmpl': '{}.apk'.format(apk_name),
'progress_hooks': [rename_hook],
'writeautomaticsub': True,
'nocheckcertificate': True,
'nooverwrites': True,
'subtitleslangs': 'en',
'logger': MyLogger(),
'retries': 3,
'socket_timeout': 60
}
with youtube_dl.YoutubeDL(ydl_opts) as ydl:
rest = ydl.download([apk_link])
if not rest:
return True, ''
else:
return False, FileNotFoundError
except Exception as e:
return False, e
Well, it’s time to say goodbye to everyone again. Creation is not easy, please give me a thumbs up and leave. Your support is the driving force for my creation. I hope to bring you more high-quality articles.