Skip to content

Conversation

Shinku-Chen
Copy link
Contributor

@Shinku-Chen Shinku-Chen commented Nov 29, 2023

when retry scrape
requestData will loss in http.NewRequest

so Seek requestData before scrape.NewRequest

req.ContentLength  always 0 and

if req.GetBody != nil && req.ContentLength == 0 {
			req.Body = NoBody
			req.GetBody = func() (io.ReadCloser, error) { return NoBody, nil }
		}

@Shinku-Chen Shinku-Chen changed the title Fix post retry loss data, Seek requestData before do scrape Fix Bug: retry post will loss data, so Seek requestData before scrape.NewRequest Nov 30, 2023
@Shinku-Chen Shinku-Chen changed the title Fix Bug: retry post will loss data, so Seek requestData before scrape.NewRequest Fix Bug: retry scrape will loss requestData , so Seek requestData before scrape.NewRequest Nov 30, 2023
@Shinku-Chen Shinku-Chen changed the title Fix Bug: retry scrape will loss requestData , so Seek requestData before scrape.NewRequest Fix Bug: retry scrape will loss requestData Nov 30, 2023
@Shinku-Chen Shinku-Chen changed the title Fix Bug: retry scrape will loss requestData Fix Bug: retry scrape will loss POST requestData Dec 25, 2023
@Shinku-Chen Shinku-Chen changed the title Fix Bug: retry scrape will loss POST requestData Fix Bug: retry scrape will lost POST requestData Dec 25, 2023
@Shinku-Chen
Copy link
Contributor Author

@WGH-

Copy link
Collaborator

@WGH- WGH- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's a good idea to limit the API to seekable bodies.

However, we might try to rewind the body if it happens to implement io.Seeker. And it might be a good idea to make Retry retry an error if doesn't.

@Shinku-Chen
Copy link
Contributor Author

是个好主意
现在已经按照需求修改了代码
请帮忙检查一下 感谢

That's a great idea.
The code has now been modified according to the requirements.
Please help me check it out, thank you.

@WGH-
Copy link
Collaborator

WGH- commented Mar 25, 2024

Otherwise, looks good

When calling Request.Retry, and request has a body (e.g. POST form or file
upload), try to rewind it.  If it's not seekable, return
ErrRetryBodyUnSeekable error.
@WGH- WGH- merged commit 4ccfe78 into gocolly:master Mar 25, 2024
@WGH-
Copy link
Collaborator

WGH- commented Mar 25, 2024

And thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants