How to disable header timeout in LXD 5.1?

I’m having problems running tar ... | lxc import - $instancename:

Error: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers

I’ve found out that there was a 30 second timeout added in LXD 5.1 and there is a pull request to increase it to 1 hour: Client: Increase header timeout from 30s to 1 hour by tomponline · Pull Request #10407 · lxc/lxd · GitHub
But that’s not enough, I need to import images which are hundreds of GB in size, which takes hours.
How can I disable this timeout?

Agreed 30s isn’t enough, but for long running processes LXD has async operations (which use websockets to monitor the progress of the operation), so the initial header response should still be <30s.

I’ve tested importing large instances that take over 30s (and indeed our own automated tests do this for VMs as well) and we don’t see the issue. I’ve also tested launching instances from images that take over 30s to download (by slowing down the network), and don’t see the issue there either.

So something else is at play here, but so far nobody on related issue golang client net/http: timeout awaiting response headers · Issue #10377 · lxc/lxd · GitHub has been able to provide the associated server logs at the same time as the client request is timing out. So I am not sure what is going on.

I am also planning to increase the timeout massively until we can figure it out Client: Increase header timeout from 30s to 1 hour by tomponline · Pull Request #10407 · lxc/lxd · GitHub, although as soon as that is merged we will likely lose visibility on the users who are experiencing these issues.

1 Like

Once merged @stgraber can cherry-pick into the LXD 5.1 release snap.

If you can reproduce this issue then it would be valuable to see the server log output from lxc monitor --type=logging --pretty at the same time as running the command that is failing with the --debug flag.

I’ve temporarily paused migrations to LXD, so I didn’t manage to get the log, but today I managed to reproduce it with a simple lxc list:

root@osg-node6:~# lxc list
Error: Get "http://unix.socket/1.0/instances?recursion=2": net/http: timeout awaiting response headers

I’ve also go the output of lxc monitor --type=logging --pretty, which is 7210 lines. Would that be useful?

Yes please

I started to go over the 7210 lines in the log to make sure I don’t leak something private, but it was taking too much time and since then LXD 5.3 was released, so I’ll try the new version.

1 Like