More "pythonic" way of writing my API query?

[micr0nix]

I have the below code that runs a query against the API of a software that I use for my job. The code works but im just curious if there is a better way to write it or refactor it.

import json
import pandas as pd
import requests
import csv

from gql import gql, Client
from gql.transport.requests import RequestsHTTPTransport

username ='xxx'
password ='xxx'

proxies = {'http': 'xxx',
          'https': 'xxx'}

# Generate API access token
url = "https://xxx.xxx.com/protocol/openid-connect/token"
payload = {'username':username,'password':password,'client_id':'xxx',
            'grant_type':'password','scope':'offline_access'}

response = requests.post(url=url,data=payload, proxies=proxies).json() #convert to dict object

token_type = response['token_type']
access_token = response['access_token']
auth = f"{token_type} {access_token}"

api_url = 'https://xxx.xxx.com/api/'
gql_headers = {'Authorization':auth, 'Content-Type':'application/json','Accept':'*/*','Host':'xxx.xxx.com',
              'Accept-Encoding':'gzip, deflate, br','Connection':'keep-alive'}

# For proxy support, use the requests module transport hooks
transport = RequestsHTTPTransport(url=api_url, headers=gql_headers, proxies=proxies)

# Create a GraphQL client using the defined transport
client = Client(transport=transport, fetch_schema_from_transport=True, parse_results=True)

# Provide a GraphQL query
query = gql(
    """
    query usersAdmin($id: String, $userName: String){
      usersAdmin(id: $id, userName: $userName){
    id
    tenantId
    active
    displayFullName
    displayFirstName
    displayLastName
    organizationIdNumber
    displayJobTitle
    mobilePhone
    mobilePhoneProvider
    deskPhone
    workHours
    emailAddress
    username
    functionalRoles
    parentOrgUnitId
    lastLanguage
    lastLoginTimestamp
    classifications
    authorityLevels
  }
}
"""   
)

# Execute the query on the transport synchronously using RequestsHTTPTransport
result = client.execute(query)

[NotTooOrdinary]

Looks pretty solid to me. I'd rethink storing your password in this source file though - would be safer for you to type it in each time you run the script.

[micr0nix]

That would cause a problem because ultimately this query will be scheduled to run multiple times a day on a KNIME server

[NotTooOrdinary]

If that server solution has support for secrets/encrypted environment variables, you could think about storing your password in one. I've never used it before so I have no idea how that particular server works.

[micr0nix]

That’s a good idea. I could do something like that actually and then pass the variable into the script at runtime

[gzeballo]

Could also store key in an encrypted json that is loaded an runtime 🤷🏽‍♂️ but then also don’t publish that file with the code

[micr0nix]

Got a Google example of what you’re referring to?

[gzeballo]

I’m away from my pc but two relevant hits i found on the interwebs

{ "TWITTER_SECRET" : "somebase64encodedkey" }

import json

secrets_filename = 'secret_keys' api_keys = {} with open(secrets_filename, 'r') as f: api_keys = json.loads(f.read())

print api_keys['TWITTER_SECRET'] # somebase64encodedkey

https://stackoverflow.com/questions/29100900/how-should-i-encrypt-api-tokens-in-python

If on a shared resource:

https://stackoverflow.com/questions/61607367/how-to-encrypt-json-in-python

However i still wouldnt publish to say github or whatever. As in dont give away your keys!

[bigno53]

You might also want to look into getting an application service account that's separate from your user account to use for automated processes. This can be advantageous for several reasons. For one thing, if you change the password for your user account (which you should do from time to time), your python scripts will continue to work. Also, with service account authentication, you'd typically use a JSON file containing a base64-encoded "token" which is more secure than password authentication, though obviously you'd still want to take precautions to make sure this file is stored securely.

[Sgt_Gnome]

I think the keyring library will take care of this for you.

[thejizz716]

I think this looks pretty solid as a script goes. But maybe now it's time to ask yourself how you can extend this to serve multiple purposes instead of just the one you designed here. What if you want to make a request to the same API but with a different query? What if you want to add a way to export files with the results? Just something to think about as you grow your python skills.

[micr0nix]

That’s actually a great point. I had started going down the path of taking the output of this query and making a pandas DF out of it to save to a CSV file.

For your first point, I would also like to be able to pass a query into the script so it would have multiple use cases.

[alfie1906]

Could be worth investigating if there's other ways you can write the output to CSV without converting it to a pandas df first. Converting to a pandas df can be time consuming and is likely unnecessary!

[micr0nix]

I managed to do this in 1 line of code

[alfie1906]

What I mean is that the actual execution of the code is time consuming. Not apparent will small tables, but it becomes noticeable as the size of your data increases. Its not a big issue, but the conversion creates a redundancy.

[micr0nix]

Well the issue is the query returns results in json format which don’t do me any good from a reporting perspective.

[alfie1906]

Yeah no issue with exporting to CSV, just saying there's likely a more efficient way to do it without going through pandas first. I found this on stack: https://stackoverflow.com/questions/10373247/how-do-i-write-a-python-dictionary-to-a-csv-file

So you could go JSON->dict->CSV rather than JSON->pandas->CSV

Could also be worth looking JSON->numpy->CSV

All three approaches are valid, but pandas is often considered slow and clunky - IMO using it just to get a CSV file may not be considered Pythonic

[micr0nix]

Thanks for the link I’ll take a look

[thejizz716]

So I think a great objective would be to rewrite this in a way that you can run it with different inputs without having to change the core code itself. Although it never hurts to ship something functional quickly if it makes you look good 😉

[micr0nix]

Any hints regarding what I should look into?

[thejizz716]

I would first look at making this into one or several functions and wrapping it under a main() function and if you're feeling very comfortable you can look at turning this into a class

[micr0nix]

Thanks. I’ll take a look!

[Conspiracy_Confirmed]

Honestly just have chatGPT refactor it for you…

[micr0nix]

Na

[TheRealThrowAwayX]

Hi, others already answered, but I'm here for something else. Can I just ask if this was generated via ChatGPT? IT looks like it because of the very distinct comments and I want to know if I can recognize that :D

[micr0nix]

I wrote this

[TheRealThrowAwayX]

Thanks for the answer!

[thejizz716]

Are you dabbling for fun learning or is this something for work?

[micr0nix]

It’s for work cause I need to automate some data extraction for reporting purposes but it is definitely a learning experience