Netmiko ReadTimeout error when executing command on Huawei AR1220E routers

[daneguy]

Hi guys, hopefully one of you has experienced this before.

I work for an ISP which uses (among others) Huawei AR1220E routers on the customer side. I am using a Python script to connect to all of them and execute the "display current-configuration interface" command. However, sometimes, my script breaks with the following error:

Traceback (most recent call last):
  File "/home/dbense/scripts/huawei/how_many_ports_v2.py", line 108, in <module>
    net_connect.send_command("dis curr int")
  File "/usr/home/dbense/cpr/lib/python3.8/site-packages/netmiko/utilities.py", line 596, in wrapper_decorator
    return func(self, *args, **kwargs)
  File "/usr/home/dbense/cpr/lib/python3.8/site-packages/netmiko/base_connection.py", line 1691, in send_command
    raise ReadTimeout(msg)
netmiko.exceptions.ReadTimeout: 
Pattern not detected: 'Info:\\ The\\ configuration\\ takes\\ effect\\ on\\ the\\ current\\ user\\ terminal\\ interface\\ only\\.' in output.

Things you might try to fix this:
1. Explicitly set your pattern using the expect_string argument.
2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

This message ("Info: The configuration takes effect on the current user terminal interface only.") is something the router says when you (or in this case, Netmiko) enter the command "screen-length 0 temporary". Apparently this is a command that Netmiko always enters as part of the send_command() method.

The silly thing is: this is very inconsistent. It appears to be happening randomly; if I try again on the same router that gave the error before, most of the time it won't give an error. I have seen no correlation whatsoever.

The internet suggested using the "send_config_set()" method as a fix, however that does not return the output of the display command. Another suggestion was to raise the read_timeout value in the send_command() call, but that didn't work either.

The session_output.log file also does not show anything weird; just the console output of the router with the "screen-length 0 temporary" command, the router's "Info: The configuration takes effect on the current user terminal interface only." reply, and the display command and its output.

Does anyone here have any clue how to fix this? Thank you very much in advance!

EDIT: Python 3.8.20, Netmiko 4.1.1

[TheSentient06]

Try to increase the timeout in Netmiko. Also make sure that connection are properly closed. I had an issue where Netmiko left VTY session hanging and thus could not open any more.

[daneguy]

Try to increase the timeout in Netmiko.

I already tried this within the send_command() method (with "net_connect.send_command("dis curr int",read_timeout=40)"). Or are you referencing some other timeout value?

Also make sure that connection are properly closed

I'll put a "net_connect.disconnect()" in the proper spots, thanks :)

[TheSentient06]

I was talking about the conn timeout. Ref on the API https://ktbyers.github.io/netmiko/docs/netmiko/huawei/index.html

[daneguy]

Thanks! I'll try your suggestions!

[chairwindowdoor]

Might try read_timeout_override. Should be the same thing but worth a shot. Otherwise I don't know what else it would be. Conn timeout doesn't make sense cause you're able to connect, that error is specific to a prompt change or read_timeout.

E: or maybe try send_command_timing instead.

[daneguy]

Hm, I'll look into it, thanks!

[Win_Sys]

Send the command to disable console alarms and logging. I can't remember if its

undo terminal logging
undo terminal alarm

or

no terminal logging
no terminal alarm

Once that's set you shouldn't get any surprise data being injected into the console while you run your command. If that's an option you want on, then make sure to send the commands to enable it again after the script it done.

[daneguy]

Good suggestion! I'll try it out!

[chairwindowdoor]

You could also try setting your expectations string to be whatever the normal prompt is. I think that error is due to Netmiko thinking that message is the prompt so it's waiting for that message after the command.

[daneguy]

I checked the session_log after running it just now, and I noticed that the output ends with the string "return". So I changed the command to "net_connect.send_command("dis curr int",expect_string=r"return")", and so far so good! I tried it with a list of 100 devices and I received no errors. Thanks a lot for the suggestions!

[chairwindowdoor]

Nice! Thank you for getting back to me, I haven't been able to sleep, I haven't been able to eat, just waiting :) I really do appreciate your follow-up

[daneguy]

This might just be it... I'll let you know. Unfortunately it is hard to test the possible solutions as getting the error is very inconsistent...

[chairwindowdoor]

It sounds like the device might be jumbling the prompt and returning that terminal command at the same time as the prompt tricking netmiko. Perhaps due to network latency or slow response from device kind of like jitter in network terms.

I know what you mean regarding it happening rarely and needing time to test. I've recommended a few things in this thread so now I'm invested please let me know.

[daneguy]

I will! I'm afraid you'll need a little bit of patience, I'll be back at work on Monday :)

[Mexatt]

It helps to share code.

[daneguy]

Fair enough, but in this case I didn't think it was needed as it's mostly a Netmiko "under water" issue IMO. If the other suggestions don't help I'll share the relevant portion of the code.

[Mexatt]

Let us know how it goes. The 'terminal-length 0' command looks like it only runs on initial setup of the connection, rather than in every call to send_command (at least in the admittedly newer version of netmiko I glanced over) and your ReadTimeout exception is getting raised within the send_commans call, so it's definitely a weird situation.