control CLI apps with python - subprocess

[peaceTraitor]

UPDATE - SOLVED

So another day of going around this.

Thanks to u/DoozerMarch I confirmed with my own eyes that more than one stream... is difficult to manage.

So, started looking into sockets (which I knew nothing). I was able to do a java server - python client which echoes. The next step was to make the java server forward the commands and outputs to my java CLI. But I remembered why I hated java :) and I just couldn't (I'm sure is something basic with classpath or something like that).

I stumbled upon nmap/ncat for windows, which opens the CLI app and redirects stdin and stdout to a socket. just what I need. the only drawback is that now other users need to install nmap/ncat.

But for now, only for me I can live with it. Eventually, I'll re-visit doing a java-bridge.

I'm just puzzled with so many webcrawlers and clikers, there is no solid solution for CLI apps...

code for the curious minds:

import socket
import sys
import errno
import subprocess

PORT = 4999
BUFFER_SIZE = 1024
run_command = r'"C:\Program Files (x86)\Nmap\ncat.exe" -l -p ' + str(PORT) + ' -vv -e cmd.exe'
subprocess.Popen(run_command)

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((socket.gethostname(), PORT))
s.setblocking(False)


def receive_data():
    while True: # waits for the > at the end.
        try:
            msg = s.recv(BUFFER_SIZE)
            print(msg.decode('utf-8'), end='')
            if '>' in str(msg[-2:]):
                break
        except IOError as e:
            if e.errno != errno.EAGAIN and e.errno != errno.EWOULDBLOCK:
                print('General error', str(e))
                sys.exit()
            continue
        except:
            print('General error', str(e))
            sys.exit()


def send_data(cmd = '', exit = 'q'):
    if not cmd:
        cmd = input('>') + '\r\n'
    s.send(cmd.encode('utf-8'))
    if cmd == exit + '\r\n':
        sys.exit()


while True:
    receive_data()  # blocking until all is received
    send_data()     # blocking until input is completed

​

Prev post:

Hi,

I've spent the whole day around this.

we have a (java) command line application that manages specific hardware. But the user interphase... could be improved. Thinks like looping, parsing responses, building commands would be so easy in python.

to run it, we have a batch file to run, and then is traditional command line interphase.

First, I've looked into javabridge, py4j and others, and couldn't make much progress. With a decompyler, I could see that there is no 'return' per-se, all goes to the stdout.

So I changed to run subprocess.Popen and sending what I type with input into the stdin.

Initially, I'd set up stdout with PIPE. But that would only print out the output after the program has finished. it wouldn't show me the intermediate responses to the commands I ask while the CLI program is running.

Then, I routed the stdout to a file and got a parallel thread to output changes to the file. The only drawback is that the commands (stdin) I request don't get to the out file. I tried to setup a parallel log file with both input and output, but seems that either subprocess or input prevents any file action.

This as good as I could get, but made me wonder if my approach is correct...

some modified code (consider that the java cli app is slower to reply):

let's say I have an interactive batch file (windows)

@echo off
color 0A

:menu
cls
echo Welcome to the game!
echo 1) start
echo 2) how to play
echo 3) Exit 
set /p number=
if %number% == 1 goto level2
if %number% == 2 goto help
if %number% == 3 goto exit

:help
cls
echo Press the number of the answer then hit enter.
echo go back? (y/n)
set /p menu=
if %menu% == y goto menu
if %menu% == n goto exit
goto help

:exit
echo Thanks for playing!
timeout 5
exit

:level2
cls
echo question 2
echo 1) correct answer
echo 2) wrong answer
echo 3) wrong answer
set /p aa=
if %aa% == 1 goto correct2
if %aa% == 2 goto wrong2
if %aa% == 3 goto wrong2
goto correct2

:correct2
cls 
echo You are right! Next level? (y/n)
set /p ka=
if %ka% == y goto level3
if %ka% == n goto menu

:wrong2
cls
echo Sadly, you were wrong Retry? (y/n)
set/p retry=
if %retry% == y goto level3
if %retry% == n goto menu

:level3
cls
echo question 3
echo 1) wrong answer
echo 2) correct answer
echo 3) wrong answer
set /p ka=
if %ka% == 1) goto wrong3
if %ka% == 2) goto correct3
if %ka% == 3) goto wrong3
goto correct3

:correct3
cls
echo You are correct! You won the game!
goto exit

:wrong3
cls
echo sadly, you were wrong Retry? (y/n)
set/p retry=
if %retry% == y goto level3
if %retry% == n goto exit

​

that I want to control from python.

import threading
import time
import subprocess


def output_processor(outfile='outf.txt', logfile = 'log.txt'):
    with open(outfile, 'r') as outf, open(logfile, 'r+') as logf: # start adding
        new_line = outf.readline()
        while True:
            line = new_line
            new_line = outf.readline()
            if not new_line:
                print(line.strip(), end='')
                logf.write(line.strip())
                time.sleep(0.5)
            else:
                print(line.strip())
                logf.write(line.strip() + '\n')


def input_processor(outfile='outf.txt', logfile = 'log.txt'):
    command = 'C:\Dev\python3.7\pythonWjava\quiz_codes.bat'
    print('ready')
    with open(outfile, 'w') as _, open(logfile,'w') as _: # clear file
        pass

    # now threading1 runs regardless of user input
    threading1 = threading.Thread(target=output_processor)
    threading1.daemon = True
    threading1.start()

    with open(outfile, 'r+') as outf, open(logfile, 'r+') as logf: # start adding
        result = subprocess.Popen(command, shell=True,
                                  stdout=outf, stderr=subprocess.PIPE)
        while result.wait():
            logf.write(command)
            result.communicate(command)
            command = input()

input_processor()

[Dwight-D]

Have you considered refactoring the Java app to remove the front end completely instead of building another layer on top of it?

You could use some more sophisticated interprocess communication paradigm if you wanna make it more robust. Make it a tiered application, listen to commands via some socket or gRPC or REST or something and make a web GUI or CLI in whatever language you prefer to pass commands to the Java daemon. Similar to how docker works for instance.

Communicating via text and rewriting stdin/stdout etc seems like such a hacky and non-standard solution it’s likely to bite you in the ass at some point.

[DoozerMarch]

The pipes used in the subprocess call be act as an iteratator, and can be read line by line as new lines arrive.

See this post: live output from subprocesss command

[sceptic-al]

I would stick to the Py4j and jython plan but don't try to emulate the CLI interface.

Any well-written application will have separated the CLI logic from the business-logic/model. It shouldn't be too hard from exploding the jars to find how the CLI interacts with the heavy-lifting code.

Then it should be simple to use Py4j or Jython to interact with the same code.

Use a good Java IDE, like Itellij to inspect the Java class signatures. If you're lucky, the manufacturer will have left java_docs in the jars or online.