1

I am running a Go binary that has an http server with systemd. I have it setup so that Caddy has a reverse-proxy to this go http server.

http: Accept error: accept tcp [::]:8002: accept4: too many open files;
dial tcp 192.85.2.4:443: socket: too many open files

When I look at the open files of the process, I get 1025 or less, though my ulimit is set to a much larger limit:

$ lsof -p 1243 | wc -l
1025

$ ulimit -Sn
200000

$ ulimit -Hn
1048576

I'm not sure if that's the problem but seems that it could be? Seems like the Go server should be spawning new goroutines or take care of that somehow.

EDIT: Here's my server script:

package main

import (
        "fmt"
        "time"
        "net/http"
)


type Config struct{}


func (c Config) testerHandler(w http.ResponseWriter, r http.Request) {
    r.Body.Close()
    time.Sleep(1 * time.Second)
    fmt.Fprintf(w, "hello\n")
}


func main() {
        c := &Config{}
        http.HandleFunc("/tester", c.testerHandler)
        fmt.Println("listening on http://127.0.0.1:8000")
        http.ListenAndServe(":8000", nil)
}

EDIT: And here's a script I use to spam my server:

package main

import (
        "log"
        "net/http"
)


func main() {
        number := 10000
        log.Printf("spamming %d numbers\n", number)
        ch := make(chan interface{})
        client := http.Client{}


    for i:=0; i<number; i++{
            go func(number int) {
                    u := "http://127.0.0.1:8000/tester"
                    rsp, err := client.Get(u)
                    if err != nil {
                            ch <- err
                            return
                    }
                    rsp.Body.Close()

                    ch <- rsp
            }(number)
    }

    var errs int
    m := make(map[int]int)

    for i:=0; i<number; i++ {
            rsp := <-ch
            switch rsp.(type) {
            case *http.Response:
                    code := rsp.(*http.Response).StatusCode
                    m[code]++
            default:
                    log.Println(rsp.(error))
                    errs++
            }
    }

    log.Println("errs:", errs)
    for k, v := range m {
            log.Printf("%d:%d\n", k, v)
    }



}
user_78361084
  • 121

3 Answers3

1

This is a problem that can be solved by modifying some parameters of your system.

I recommend that you follow the following thread: Too many open files

Carlos Dagorret
  • 636
0

I was able to fix this by either Disabling Keep Alives on the ReverseProxy or by setting a low "ReadTimeout" on the Go Server.

Simple example

func main(){
srv := &http.Server{
    Addr: ":8080",
    ReadTimeout:    2 * time.Second,
    Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        go fmt.Printf("handling request\n")
        fmt.Fprintln(w, "Hello, client")
    }),
}
log.Fatal(srv.ListenAndServe())
}
user1270949
  • 1
0

Systemd takes over ulimits and sets the actual (=soft) runtime limit for max number of files to 1024. It is expected that the process takes care of requesting resources it needs explicitly - that is how most "professional" applications are written. For home grown ones, the resource can be requested in the dot-service file, e.g.,:

[Service]
LimitNOFILE=10240
LimitNOFILESoft=10240
SVUser
  • 121