一个过滤网站的采集

clalj520 · 发表于 2019-4-26 16:22:03

http://www.sxzhongrun.cn/news_list.php?lanmu=25&c_id=13
这个网站直接打开需要执行一个js
按抓包的内容采集不到真实页面只是js代码,下面是抓包

Request URL:
http://www.sxzhongrun.cn/news_list.php?lanmu=25&c_id=13
Request Method:
GET
Status Code:

200 OK
Remote Address:
222.171.249.201:80

Response Headersview source
- Cache-Control:
  no-store, no-cache, must-revalidate, post-check=0, pre-check=0
- Connection:
  close
- Content-Type:
  text/html;charset=utf-8
- Date:
  Fri, 26 Apr 2019 08:15:38 GMT
- Expires:
  Thu, 19 Nov 1981 08:52:00 GMT
- Pragma:
  no-cache
- Server:
  IIS
- X-Powered-By:
  WAF/2.0
- X-Powered-By:
  WAF/2.0
Request Headersview source
- Accept:
  text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
- Accept-Encoding:
  gzip, deflate, sdch
- Accept-Language:
  zh-CN,zh;q=0.8
- AlexaToolbar-ALX_NS_PH:
  AlexaToolbar/alxg-3.3
- Connection:
  keep-alive
- Cookie:
  safedog-flow-item=4BB79AF79A46646D16D11848D15ED48A; PHPSESSID=ur4aeho1pade07bas0rve8etv4; UM_distinctid=16a58adc713c2-0630c398f15345-d6a580b-100200-16a58adc714773; CNZZDATA1253261798=1571369698-1556265942-http%253A%252F%252Fwww.sxzhongrun.cn%252F%7C1556265942
- DNT:
  1
- Host:
  www.sxzhongrun.cn
- Referer:
  http://www.sxzhongrun.cn/news_list.php?lanmu=25&c_id=13&WebShieldSessionVerify=ZSxGN4EDUJGxK92P0rOh
- Upgrade-Insecure-Requests:
  1
- User-Agent:
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2900.0 Iron Safari/537.36
Query String Parametersview sourceview URL encoded
item = new HttpItem()
{
URL = "http://www.sxzhongrun.cn/news_list.php?lanmu=25&c_id=13",//URL
Method = "get",//URL 可选项默认为Get
Referer = "http://www.sxzhongrun.cn/news_list.php?lanmu=25&c_id=13&WebShieldSessionVerify=ZSxGN4EDUJGxK92P0rOh",
Cookie = "safedog-flow-item=4BB79AF79A46646D16D11848D15ED48A; PHPSESSID=ur4aeho1pade07bas0rve8etv4; UM_distinctid=16a58adc713c2-0630c398f15345-d6a580b-100200-16a58adc714773; CNZZDATA1253261798=1571369698-1556265942-http%253A%252F%252Fwww.sxzhongrun.cn%252F%7C1556265942",
Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2900.0 Iron Safari/537.36"
};
HttpResult result = http.GetHtml(item);

clalj520 · 发表于 2019-4-26 16:23:21

最下是我写的代码!这样获取不到采集的页面!请大虾帮忙看下!

站长苏飞 · 发表于 2019-4-27 10:14:42

http://www.sxzhongrun.cn/news_list.php?lanmu=25&c_id=13 用我的框架直接请求就行啊。能取到内容

站长苏飞 · 发表于 2019-4-27 10:15:06

http://tool.sufeinet.com/HttpHel ... anmu=25&c_id=13

clalj520 · 发表于 2019-5-8 16:18:47

要怎么弄?我弄了几次都是没抓到需要的数据
通过
http://tool.sufeinet.com/HttpHel ... anmu=25&c_id=13
设定Header 总显示未将对象引用设置到对象的实例。

clalj520 · 发表于 2019-5-8 16:21:57

view plainprint?
返回的Header：
Connection:Close
Pragma:no-cache
Content-Length:435
Cache-Control:no-store
Content-Type:text/html; Charset=gb2312
Date:Wed, 08 May 2019 08:21:57 GMT
Server:IIS


返回的Cookie：


最后访问URL：http://www.sxzhongrun.cn/news_list.php?lanmu=25&c_id=13

301跳转URL：

返回的HTML：
<html><head><meta http-equiv="Content-Type" content="text/html; charset=gb2312" /><meta http-equiv="pragma" content="no-cache" /><meta http-equiv="cache-control" content="no-store" /><meta http-equiv="Connection" content="Close" /><script>function JumpSelf(){ self.location="/news_list.php?lanmu=25&c_id=13&WebShieldSessionVerify=vpjXA6tzKVZS48FCSRkx";}</script><script>setTimeout("JumpSelf()",700);</script></head><body></body></html>

clalj520 · 发表于 2019-5-11 16:59:27

求飞哥指教下!

		自动登录	找回密码
密码			马上注册

[求助] 一个过滤网站的采集