设为首页收藏本站
开启辅助访问 广告合作开通会员官方QQ/微信群新人报到处
http://www.sufeinet.com/plugin.php?id=keke_group

苏飞论坛

 找回密码
 马上注册

QQ登录

只需一步，快速开始

分布式系统框架(V2.0) 轻松承载百亿数据，千万流量！讨论专区 - 源码下载 - 官方教程

HttpHelper爬虫框架(V2.7-含.netcore) HttpHelper官方出品，爬虫框架讨论区 - 源码下载 - 在线测试和代码生成

HttpHelper爬虫类(V2.0) 开源的爬虫类,支持多种模式和属性 源码 - 代码生成器 - 讨论区 - 教程- 例子

苏飞论坛»主页 › 苏飞工作室作品 › 网络爬虫类HttpHelper › [C#HttpHelper]类DotNet4.0正式版教程与升级报告
12
返回列表 发新帖
楼主: 站长苏飞

[教程] [C#HttpHelper]类DotNet4.0正式版教程与升级报告

[复制链接]
站长苏飞
 楼主| 发表于 2013-8-15 14:25:25 | 显示全部楼层
若如初见 发表于 2013-8-15 13:00
有个问题想说一下：
1、一般用完的HttpWebRequest要Abort()或者要把 Response.Close()，httphelper里面没 ...

1.你这点说的很对，只不过我里面写的是用的，只是语法不一样
[code=csharp] using (response = (HttpWebResponse)request.GetResponse())[/code]
看到这里你就明白了，如果不清楚什么是using 语句可以百度一下
2.httpitem.cookie和httpitem.cookiecollection不是只有一个生效，而是看你使用那个，那个就有效
这个取决于服务端。
3.你再看下类面的这句 if (response.Cookies != null) result.CookieCollection = response.Cookies;
这和你用什么容器接收没有关系，所以完全可以排除你说的情况，这和使用cookiecontainer还是CookieCollection接收没有任何关系的。
肯定是其他地方影响的。
4. if (item.ProtocolVersion != null) request.ProtocolVersion = item.ProtocolVersion;你看看代码，如果你不设置是不会赋值，也就是默认的，所以这里你说的情况不存在，
非常 建议您再好好看下我的源码。同时对你提出的建议表示感谢





1. 开通SVIP会员，免费下载本站所有源码，不限次数据，不限时间
2. 加官方QQ群，加官方微信群获取更多资源和帮助
3. 找站长苏飞做网站、商城、CRM、小程序、App、爬虫相关、项目外包等点这里
回复 支持 反对

使用道具 举报

若如初见
发表于 2013-8-16 11:50:11 | 显示全部楼层
站长苏飞 发表于 2013-8-15 14:25
1.你这点说的很对，只不过我里面写的是用的，只是语法不一样

看到这里你就明白了，如果不清楚什么是us ...

首先感谢您的回复
第一个问题抱歉，是我看源码不仔细，后面几个问题我把代码敲出来验证一下，
再次麻烦您很抱歉，我很喜欢httphelper类，我只想解决心中的疑惑，也希望httphelper越做越好，所以麻烦您看一下我的代码：
首先看这个代码
  1. private void button1_Click(object sender, EventArgs e)
  2. {
  3. HttpItem loginItem= new HttpItem();
  4. loginItem.UserAgent = "Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16";
  5. loginItem.Method = "post";
  6. loginItem.URL = "https://account.xiaomi.com/pass/serviceLoginAuth";
  7. loginItem.Postdata = "user=axiaomi007%40163.com&pwd=xiaomi8513&callback=http%3A%2F%2Fm.xiaomi.com%2Fmshopapi%2Fv1%2Fauthorize%2Fsso_callback%3Ffollowup%3Dhttp%253A%252F%252Fm.xiaomi.com%252F%2523ac%253Daccount%2526op%253Dindex%26sign%3DNGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ%2C%2C&sid=eshopm&display=mobile";
  8. loginItem.ContentType = "application/x-www-form-urlencoded";
  9. loginItem.Allowautoredirect = true;

  11. HttpHelper httphelper = new HttpHelper();
  12. HttpResult result = httphelper.GetHtml(loginItem);

  14. textBox1.Text = result.Html;
  15. }
复制代码
执行后textBox1的内容为：“基础连接已经关闭: 接收时发生错误。”；
然后在上面加上
loginItem.SecurityProtocolType = SecurityProtocolType.Ssl3;
之后就能正常接收了，
说明SecurityProtocolType的默认值是不行的，所以建议设置默认值
回复 支持 反对

使用道具 举报

若如初见
发表于 2013-8-16 11:53:00 | 显示全部楼层
站长苏飞 发表于 2013-8-15 14:25
1.你这点说的很对，只不过我里面写的是用的，只是语法不一样

看到这里你就明白了，如果不清楚什么是us ...

再说cookie问题
  1. private void button1_Click(object sender, EventArgs e)
  2. {
  3.   HttpItem loginItem= new HttpItem();
  4.   loginItem.UserAgent = "Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16";
  5.   loginItem.Method = "post";
  6.   loginItem.URL = "https://account.xiaomi.com/pass/serviceLoginAuth";
  7.   loginItem.Postdata = "user=axiaomi007%40163.com&pwd=xiaomi8513&callback=http%3A%2F%2Fm.xiaomi.com%2Fmshopapi%2Fv1%2Fauthorize%2Fsso_callback%3Ffollowup%3Dhttp%253A%252F%252Fm.xiaomi.com%252F%2523ac%253Daccount%2526op%253Dindex%26sign%3DNGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ%2C%2C&sid=eshopm&display=mobile";
  8.   loginItem.ContentType = "application/x-www-form-urlencoded";
  9.   loginItem.Allowautoredirect = true;
  10.   loginItem.SecurityProtocolType = SecurityProtocolType.Ssl3;

  12.   HttpHelper httphelper = new HttpHelper();
  13.   HttpResult result = httphelper.GetHtml(loginItem);

  15.   textBox1.Text = result.Html;

  17.   if (result.CookieCollection != null)
  18.     MessageBox.Show(result.CookieCollection.Count.ToString());
  19.   else MessageBox.Show("空");
  20. }
复制代码
执行结果messagebox显示0
抓包数据如下：
首先是post请求：
POST /pass/serviceLoginAuth HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16
Host: account.xiaomi.com
Content-Length: 300
Expect: 100-continue
Connection: Keep-Alive
user=axiaomi007%40163.com&pwd=xiaomi8513&callback=http%3A%2F%2Fm.xiaomi.com%2Fmshopapi%2Fv1%2Fauthorize%2Fsso_callback%3Ffollowup%3Dhttp%253A%252F%252Fm.xiaomi.com%252F%2523ac%253Daccount%2526op%253Dindex%26sign%3DNGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ%2C%2C&sid=eshopm&display=mobile

响应是一个跳转，有多个setcookie：
HTTP/1.1 302 Found
Server: Server/1.4.6
Date: Fri, 16 Aug 2013 03:09:18 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 414
Connection: keep-alive
extension-pragma: {"ssecurity":"Ai05RsHfuvac5zCzn7TIjQ==","nonce":4584575710089725952,"psecurity":"VnspBo1t0gZEOjcPmRgzkA=="}
Location: http://m.xiaomi.com/mshopapi/v1/authorize/sso_callback?followup=http%3A%2F%2Fm.xiaomi.com%2F%23ac%3Daccount%26op%3Dindex&sign=NGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ,,&auth=GOmpdbc1mTbhiuq0A8Mb3BtpwuuEUZli%2FF%2F0eZuQsW8lOGfVG8p6bfB4b%2B1tS9YAvWQsuHbzuUUpck86pRKanqhZ%2BvVSzn73%2FkR20xkeVCWX42d5H8EuREntWvS4tuaSlu1%2Bbxxi07hnpQ70KkHki99ur%2FJd3FUxyM8GXM4apks%3D
Set-Cookie: userId=94164298; domain=.xiaomi.com; path=/
Set-Cookie: userId=94164298; path=/
Set-Cookie: pExpireTime=-1; path=/; secure
Set-Cookie: passToken=0QxjFrjieRMkJ7AH4gTZJE3Aqn9rwdLyRFBqPHQ0AluA5EpCUVyl1f8uW5trmFFPPAtB6o4Ib6ft4V4DEvH9g7uGeiSUI2rvOGqyYzBbLKnrIZDqpRW3fxx/js0/i2L9pj0UK8790eX3EkXJwRsNMrk67VUDI+TAD/ITCX2miNWO9zxRqnqXMrQUVVwcqY+7; path=/
Set-Cookie: uLocale=zh_CN; path=/; expires=Sat, 31-Aug-2013 03:09:17 GMT
The URL has moved <a href="http://m.xiaomi.com/mshopapi/v1/authorize/sso_callback?followup=http%3A%2F%2Fm.xiaomi.com%2F%23ac%3Daccount%26op%3Dindex&sign=NGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ,,&auth=GOmpdbc1mTbhiuq0A8Mb3BtpwuuEUZli%2FF%2F0eZuQsW8lOGfVG8p6bfB4b%2B1tS9YAvWQsuHbzuUUpck86pRKanqhZ%2BvVSzn73%2FkR20xkeVCWX42d5H8EuREntWvS4tuaSlu1%2Bbxxi07hnpQ70KkHki99ur%2FJd3FUxyM8GXM4apks%3D">here</a>

然后是自动跳转的请求，自动跳转的请求理应带上上面的cookie信息，但是却没有：
GET /mshopapi/v1/authorize/sso_callback?followup=http%3A%2F%2Fm.xiaomi.com%2F%23ac%3Daccount%26op%3Dindex&sign=NGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ,,&auth=GOmpdbc1mTbhiuq0A8Mb3BtpwuuEUZli%2FF%2F0eZuQsW8lOGfVG8p6bfB4b%2B1tS9YAvWQsuHbzuUUpck86pRKanqhZ%2BvVSzn73%2FkR20xkeVCWX42d5H8EuREntWvS4tuaSlu1%2Bbxxi07hnpQ70KkHki99ur%2FJd3FUxyM8GXM4apks%3D HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16
Host: m.xiaomi.com
Connection: Keep-Alive

然后是对应的响应，可以看到由于没有携带cookie而跳回首页：
HTTP/1.1 302 Moved Temporarily
Server: MIWS
Date: Fri, 16 Aug 2013 03:09:17 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: serviceToken=a9bJC3bAa%2BDNHmgfIr8%2BEGNd1C%2B7GNTngIJ9Gubi%2F%2FMsa7138%2BlVYktJ%2FDVN2vM661k3sbEu9zyRXN4fMIeuZ8Ycwrvz6BZ3DRNDnDrumKehDI%2BtUn6H94Vj2lx3GprETiSM8UNRYst2uK02bSaT2BkrO%2ByPuCXfsT0V8I7XIIQ%3D; path=/; domain=.m.xiaomi.com
Set-Cookie: ticketToken=a9bJC3bAa%2BDNHmgfIr8%2BEGNd1C%2B7GNTngIJ9Gubi%2F%2FMsa7138%2BlVYktJ%2FDVN2vM661k3sbEu9zyRXN4fMIeuZ8Ycwrvz6BZ3DRNDnDrumKehDI%2BtUn6H94Vj2lx3GprETiSM8UNRYst2uK02bSaT2BkrO%2ByPuCXfsT0V8I7XIIQ%3D; path=/; domain=.xiaomi.com
Set-Cookie: mishoptag=GNA4NUT1vrK9p5n1sc3CqPb1rt7tM6H8z7uTBU1T%2BXc%3D; path=/; domain=.m.xiaomi.com
location: http://m.xiaomi.com/#ac=account&op=index
Cache-Control: must-revalidate,no-cache
Vary: Accept-Encoding
0

跳转到首页的抓包就不贴上来了。。。
然后接着说代码：
把自动跳转关闭测试
loginItem.Allowautoredirect = false;
相当于上面的抓包信息只执行了第一步请求和响应，响应里面有set_cookie;
然后MessageBox.Show(result.CookieCollection.Count.ToString());显示依然为0，说明响应的cookie就没有添加到cookiecollection里面
回复 支持 反对

使用道具 举报

若如初见
发表于 2013-8-16 11:56:43 | 显示全部楼层
站长苏飞 发表于 2013-8-15 14:25
1.你这点说的很对，只不过我里面写的是用的，只是语法不一样

看到这里你就明白了，如果不清楚什么是us ...

然后说cookie和cookiecollection只有一个生效的问题
在loginItem.CookieCollection添加一个名为test1的cookie测试
  1. private void button1_Click(object sender, EventArgs e)
  2. {
  3.   HttpItem loginItem= new HttpItem();
  4.   loginItem.UserAgent = "Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16";
  5.   loginItem.Method = "post";
  6.   loginItem.URL = "https://account.xiaomi.com/pass/serviceLoginAuth";
  7.   loginItem.Postdata = "user=axiaomi007%40163.com&pwd=xiaomi8513&callback=http%3A%2F%2Fm.xiaomi.com%2Fmshopapi%2Fv1%2Fauthorize%2Fsso_callback%3Ffollowup%3Dhttp%253A%252F%252Fm.xiaomi.com%252F%2523ac%253Daccount%2526op%253Dindex%26sign%3DNGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ%2C%2C&sid=eshopm&display=mobile";
  8.   loginItem.ContentType = "application/x-www-form-urlencoded";
  9.   loginItem.Allowautoredirect = false;
  10.   loginItem.SecurityProtocolType = SecurityProtocolType.Ssl3;

  12.   loginItem.CookieCollection = new CookieCollection();//添加test1 的cookie
  13.   loginItem.CookieCollection.Add(new Cookie("test1", "test1", "/", ".xiaomi.com"));

  15.   HttpHelper httphelper = new HttpHelper();
  16.   HttpResult result = httphelper.GetHtml(loginItem);
  17. }
复制代码
请求数据包如下
POST /pass/serviceLoginAuth HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16
Host: account.xiaomi.com
Cookie: test1=test1
Content-Length: 300
Expect: 100-continue
user=axiaomi007%40163.com&pwd=xiaomi8513&callback=http%3A%2F%2Fm.xiaomi.com%2Fmshopapi%2Fv1%2Fauthorize%2Fsso_callback%3Ffollowup%3Dhttp%253A%252F%252Fm.xiaomi.com%252F%2523ac%253Daccount%2526op%253Dindex%26sign%3DNGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ%2C%2C&sid=eshopm&display=mobile

在loginItem.Cookie添加一个名为test2的cookie测试
  1. private void button1_Click(object sender, EventArgs e)
  2. {
  3.   HttpItem loginItem= new HttpItem();
  4.   loginItem.UserAgent = "Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16";
  5.   loginItem.Method = "post";
  6.   loginItem.URL = "https://account.xiaomi.com/pass/serviceLoginAuth";
  7.   loginItem.Postdata = "user=axiaomi007%40163.com&pwd=xiaomi8513&callback=http%3A%2F%2Fm.xiaomi.com%2Fmshopapi%2Fv1%2Fauthorize%2Fsso_callback%3Ffollowup%3Dhttp%253A%252F%252Fm.xiaomi.com%252F%2523ac%253Daccount%2526op%253Dindex%26sign%3DNGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ%2C%2C&sid=eshopm&display=mobile";
  8.   loginItem.ContentType = "application/x-www-form-urlencoded";
  9.   loginItem.Allowautoredirect = false;
  10.   loginItem.SecurityProtocolType = SecurityProtocolType.Ssl3;

  12.   loginItem.Cookie = "test2=test2";

  14.   HttpHelper httphelper = new HttpHelper();
  15.   HttpResult result = httphelper.GetHtml(loginItem);
  16. }
复制代码
请求数据包如下：
POST /pass/serviceLoginAuth HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16
Cookie: test2=test2
Host: account.xiaomi.com
Content-Length: 300
Expect: 100-continue
Connection: Keep-Alive
user=axiaomi007%40163.com&pwd=xiaomi8513&callback=http%3A%2F%2Fm.xiaomi.com%2Fmshopapi%2Fv1%2Fauthorize%2Fsso_callback%3Ffollowup%3Dhttp%253A%252F%252Fm.xiaomi.com%252F%2523ac%253Daccount%2526op%253Dindex%26sign%3DNGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ%2C%2C&sid=eshopm&display=mobile

同时添加test1和test2,代码如下
  1. private void button1_Click(object sender, EventArgs e)
  2. {
  3.   HttpItem loginItem= new HttpItem();
  4.   loginItem.UserAgent = "Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16";
  5.   loginItem.Method = "post";
  6.   loginItem.URL = "https://account.xiaomi.com/pass/serviceLoginAuth";
  7.   loginItem.Postdata = "user=axiaomi007%40163.com&pwd=xiaomi8513&callback=http%3A%2F%2Fm.xiaomi.com%2Fmshopapi%2Fv1%2Fauthorize%2Fsso_callback%3Ffollowup%3Dhttp%253A%252F%252Fm.xiaomi.com%252F%2523ac%253Daccount%2526op%253Dindex%26sign%3DNGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ%2C%2C&sid=eshopm&display=mobile";
  8.   loginItem.ContentType = "application/x-www-form-urlencoded";
  9.   loginItem.Allowautoredirect = false;
  10.   loginItem.SecurityProtocolType = SecurityProtocolType.Ssl3;

  12.   loginItem.CookieCollection = new CookieCollection();
  13.   loginItem.CookieCollection.Add(new Cookie("test1", "test1", "/", ".xiaomi.com"));

  15.   loginItem.Cookie = "test2=test2";

  17.   HttpHelper httphelper = new HttpHelper();
  18.   HttpResult result = httphelper.GetHtml(loginItem);
  19. }
复制代码
请求数据包如下，可以看到请求数据包里面只有test1,而test2则没有生效：
POST /pass/serviceLoginAuth HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16
Host: account.xiaomi.com
Cookie: test1=test1
Content-Length: 300
Expect: 100-continue
Connection: Keep-Alive
user=axiaomi007%40163.com&pwd=xiaomi8513&callback=http%3A%2F%2Fm.xiaomi.com%2Fmshopapi%2Fv1%2Fauthorize%2Fsso_callback%3Ffollowup%3Dhttp%253A%252F%252Fm.xiaomi.com%252F%2523ac%253Daccount%2526op%253Dindex%26sign%3DNGNmM2VmZDlkZTQ5MGMzMmNjN2Y2MjBmNjc0MGNiMWQxZjg0MzAxNQ%2C%2C&sid=eshopm&display=mobile
回复 支持 反对

使用道具 举报

站长苏飞
 楼主| 发表于 2013-8-16 12:10:42 | 显示全部楼层
若如初见 发表于 2013-8-16 11:56
然后说cookie和cookiecollection只有一个生效的问题
在loginItem.CookieCollection添加一个名为test1的c ...

1.默认值的问题，我建议还是不要添加的好。保存原生。呵呵。这个毕竟有待考量
2.你说的Cookie，从理论上讲。NEt返回的就是一个CookieCollection集合，你修改成cookiecontainer
无非也是将他转化一下，不可能影响他的传输的，
这点我也做过测试。不能光考虑Net环境。在有些情况下，还是比较复杂的。
在没有得到精确的结论之前我还需要先保持现在的情况，毕竟现在是原生的，没有进行任何处理，
关于这点，我有时间会有研究一下，
如果你有比较精确的结论也欢迎提出来，可以发个帖子出来。
或者加我的Qq讨论一下。
再次对你提出的建议表示感谢
这个东西官方并没有给出明显的说明，
不过建议还是使用StrCookie，
回复 支持 反对

使用道具 举报

zizhu123
发表于 2015-5-12 09:31:09 | 显示全部楼层
if (!string.IsNullOrWhiteSpace(item.Host))
            {
                request.Host = item.Host;
            }
这句代码在.net 3.5 当中该怎么写啊
回复 支持 反对

使用道具 举报

12
返回列表 发新帖
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 马上注册

本版积分规则

QQ|手机版|小黑屋|手机版|联系我们|关于我们|广告合作|苏飞论坛 ( 豫ICP备18043678号-2)

GMT+8, 2024-11-22 14:26

© 2014-2021

快速回复 返回顶部 返回列表