Golang将大文件一次读入内存

利用Slice切片,比每次读取一小块导致频繁Gc

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
package main

import (
    "bufio"
    "container/list"
    "fmt"
    "io"
    "os"
    "sync"
    "time"
)

func main(){
    start := time.Now()
    content, err := ioutil.ReadFile("d:\\software\\cn_windows_10_consumer_editions_version_21h1_x64_dvd_1baf479d.iso")
    if err != nil {
        fmt.Println("file read fail")
    }
    fmt.Printf("read file len :%d, time cost :%v slices: %d\n", len(content), time.Since(start), len(content)/(8964*20))
    step := 8964 * 20
    fmt.Println(len(content)%step, step)

    for i := 0; i < len(content); i += step {
        if i+step > len(content) {
            slice := make([]byte, len(content)%step)
            copy(slice, content[i:i+len(slice)])
            fmt.Printf("slice addr :%p slice len: %d\n", slice, len(slice))
        } else {
            slice := make([]byte, step)
            copy(slice, content[i:i+step])
            fmt.Printf("slice addr :%p slice len: %d\n", slice, len(slice))
        }
   }
}

💡 Tips: 一次将文件读入内存,用slice来遍历文件,slice本身只是引用,不会复制byte array,所以很快,而且不占用内存,减少IO,GC,提速到9000mbps