Skip to content

performance is not expected for prefetch reader #59495

@lance6716

Description

@lance6716

In current code we don't fully make use of the prefetch buffer. There's a background goroutine (spawned in line 48) that uses the prefetch buffer to read data from reader (line 57)

go ret.run()
return ret
}
func (r *Reader) run() {
defer r.wg.Done()
for {
r.bufIdx = (r.bufIdx + 1) % 2
buf := r.buf[r.bufIdx]
n, err := r.r.Read(buf)
buf = buf[:n]
select {
case <-r.closedCh:
return
case r.bufCh <- buf:
}

However if the reader only return partial data (like it's a socket and OS chooses to return only few bytes, naming N) this reading action is finished and the loop is waiting on sending to channel (line 62). If the caller of "prefetch reader" doesn't consume the channel quickly, we are wasting a large proportion of the prefetch buffer which is buf[N:] because they are not filled. We should let the background goroutine use io.ReadFull to fully use the prefetch buffer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.component/ddlThis issue is related to DDL of TiDB.severity/moderatetype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions