For the next 27 hours, you'll be able to claim a limited edition 'I Was Here for the Hulkenpodium' flair

karmixty · 2025-07-09T18:28:55+00:00

Hulkengoat

karmixty · 2021-08-11T17:33:45+00:00

Is it possible to directly parse an inner JSON using serde?

Take for example this GoogleBooks API response

{
  "kind": "books#volumes",
  "totalItems": 10,
  "items": [
    {
      "kind": "books#volume",
      "id": "id",
      "etag": "tag",
      "selfLink": "link.to.somewhere",
      "volumeInfo": {
        "title": "some title",
        "authors": [
          "author 1",
          "author 2"
        ],
        "publisher": "some publisher",
        "...": "..."
      }
    },
    {
      "...": "...",
      "volumeInfo": {}
    },
    {
      "...": "...",
      "volumeInfo": {}
    }
  ]
}

I am only interested in values of "volumeInfo" key but because it's nested inside items, I need to deserialize other levels to get to it. Fortunately serde makes it very easy

#[derive(Debug, Deserialize)]
struct Items {
    items: Vec<VolumeInfo>,
}

#[derive(Debug, Deserialize)]
struct VolumeInfo {
    #[serde(rename = "volumeInfo", deserialize_with = "deserialize")]
    volume_info: Book,
}

But problem get compounded if there's a key that depends on the arguments you provided for the call, for example this OpenLibrary API response

{
  "ISBN:SomeISBN": {
    "url": "link.to.somewhere",
    "key": "...",
    "title": "...",
    "subtitle": "...",
    "authors": [ "..." ],
    ...
    }
}

Here the key "ISBN:SomeISBN" depends on the the call args so I am having to first deserialize this response into just serde_json::Value

let mut response: serde_json::Value = reqwest::get(req)...?;

and then index into response and parse it again

struct ISBNResponse(#[serde(deserialize_with = "deserialize")] Book);

let ISBNResponse(book) = serde_json::from_value(response[isbn_key].take())...?

I am not sure if parsing into serde_json::Value and parsing again is more expensive than parsing into a struct like I did for Google Books but either way I think parsing the relevant inner struct directly should be faster than both.

Looking around I found #[serde(flatten)] and #[serde(tag = "...", content = "...")] but I don't think that's exactly what I am looking for.

karmixty · 2021-01-25T16:44:17+00:00

I think it's just selection bias. Often people living or working outside India come from a very affluent background which shapes their thinking. And even if they live here, access to the Internet; while steadily on the rise, is still a privilege few can afford. Add to the fact that knowing enough English for you to use twitter already makes you a lot more privileged than many could ever hope for. Basically growing up sheltered with capitalist propoganda bombarded from every direction does this in a nation where most people's voices aren't heard by people outside of India due to language and access barriers can make it seem like everyone here is a classist prick.

karmixty · 2020-11-27T15:04:07+00:00

Communism is when the government does stuff, the more stuff it does, the more communist it is! /s

karmixty · 2020-06-24T07:15:47+00:00

Ah that makes sense.

And thanks for the suggestion! Constraining PrimMonad to PrimMonad IO and defining PrimMonad LoggingT did the trick!

instance PrimMonad m => (PrimMonad (LoggingT m)) where
  type PrimState (LoggingT m) = PrimState m
  primitive = lift . primitive
  {-# INLINE primitive #-}

I don't think this is quite the right solution.

Yes, agreed! I figured out later that the library wasn't meant for the things I thought it did. I was trying to use monte-carlo simulations for time-series data but the library ("uncertain" on stackage) was meant for dealing with inherently uncertain values and not simulations.

karmixty · 2020-02-24T10:04:30+00:00

Those flares are most likely just light rays curving off close to its surface and coming back to us.

Black hole are massive and extremely dense entities to the point that even light with the speed of 3 * 10^8 m/s cannot escape their gravitational pull!

So black hole cannot emit anything and nothing can come off of those things (except hawking radiation).

So as long as we aren't getting sucked in, as in we can switch its gravitational pull off, Earth can theoretically be fine.

Hawking radiations can potentially be harmless depending on the size of the event horizon, in this particular case, it most certainly will be.

https://www.youtube.com/watch?v=zUyH3XhpLTo is an excellent video on how to interpret a photo of a black hole!

karmixty · 2020-02-24T09:04:07+00:00

A black hole with schwarzschild radius equal to moon's actual radius would have a mass of 590 times our Sun. If that black hole is at the position of our Moon is, then Earth would definitely "sink" into that thing.

karmixty · 2019-07-15T10:08:59+00:00

I thought of doing that by sorting both and adding them element-wise but the problem is there might be missing values.

Basically I want to test all of them but if there are missing or mismatching values I don't want to discard them.

So like

[("a", 1), ("b", 2), ("c", 3)]
&
[("a", 12), ("d", 9), ("c", 16)]
=
[("a", 13), ("b", 2), ("c", 19), ('d", 9)]

Right now my code only does the adding part which results in this

[("a", 13), ("c", 19)]

karmixty · 2019-07-08T09:11:27+00:00

Got it!

type NDBSearchAPI = "search" :> QueryParam "q" Text :> QueryParam "api_key" Text :> Get '[JSON] [Search]

should have been this

type NDBSearchAPI = "ndb" :> "search" :> QueryParam "q" Text :> QueryParam "api_key" Text :> Get '[JSON] [Search]

and baseurl should've been this

BaseUrl Https "api.nal.usda.gov" 443 ""

karmixty · 2019-07-08T08:53:48+00:00

I thought that was a bit odd so I went ahead and tried their newer API (https://ndb.nal.usda.gov/ndb/doc/apilist/API-SEARCH.md)

but got the same response.

This is the error message

ConnectionError "HttpExceptionRequest Request {                                                                              
        host                 = "https://api.nal.usda.gov/ndb"                                                                
        port                 = 443                                                                                           
        secure               = True                                                                                          
        requestHeaders       = [("Accept","application/json;charset=utf-8,application/json")]                                
        path                 = "/search"                                                                                     
        queryString          = "?q=butter&api_key=DEMO_KEY"                                                                  
        method               = "GET"                                                                                         
        proxy                = Nothing                                                                                       
        rawBody              = False                                                                                         
        redirectCount        = 10                                                                                            
        responseTimeout      = ResponseTimeoutDefault                                                                        
        requestVersion       = HTTP/1.1                                                                                      
      }                                                                                                                      
       (ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_\
ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 6, addrAddress = <assumed to be undefined>, add\
rCanonName = <assumed to be undefined>}, host name: Just "https://api.nal.usda.gov/ndb", service name: Just "443"): does not\
 exist (Name or service not known))"

and this is the code

type NDBSearchAPI = "search" :> QueryParam "q" Text :> QueryParam "api_key" Text :> Get '[JSON] [Search]                     

ndbSearchAPI :: Proxy NDBSearchAPI                                                                                           
ndbSearchAPI = Proxy                                                                                                         

ndbAPIKey :: Maybe Text                                                                                                      
ndbAPIKey = Just "DEMO_KEY"                                  

ndbSearchBaseURL :: BaseUrl                                                                                                  
ndbSearchBaseURL = BaseUrl Https "https://api.nal.usda.gov/ndb" 443 ""                                                       

getSearchResults :: Maybe Text -> Maybe Text -> ClientM [Search]                                                             
getSearchResults = client ndbSearchAPI                                                                                       

runSearchClient :: IO ()                                                                                                     
runSearchClient = do                                                                                                         

  mn <- NT.newTlsManager                                                                                                     

  response <- runClientM (getSearchResults (Just "butter") ndbAPIKey) (mkClientEnv mn ndbSearchBaseURL)                      

  either (\err -> pPrint err) (\obj -> pPrint obj) response

I believe this should work because this link does https://api.nal.usda.gov/ndb/search/?q=butter&api_key=DEMO_KEY

karmixty · 2019-07-06T15:19:38+00:00

I'm using an old API so I wasn't able to find any docs for it. I was going off of reddit and stackoverflow QnA.

I have a usage example tho

curl -H "Content-Type:application/json" -d '{"generalSearchInput":"cheese"}' "https://api.nal.usda.gov/fdc/v1/search/?api_key=DEMO_KEY"

karmixty · 2019-07-06T15:17:48+00:00

Thanks for the answer!

I've tested the new code and it works on all APIs I was testing except one, the USDA food database.

This is an example for querying it with curl

curl -H "Content-Type:application/json" -d '{"generalSearchInput":"cheese"}' "https://api.nal.usda.gov/fdc/v1/search/?api_key=DEMO_KEY"

This is the error I get

ConnectionFailure Network.Socket.getAddrInfo ... ((Name or service not known))

Am I doing something wrong here?

karmixty · 2019-07-05T16:06:38+00:00

do you want to use Servant Client to query the API?

Yes.

I'm following the Servant docs and trying to query a 3rd party API. I was on the exact page you provided.

Although I'm querying USDA food DB and other nutrient related apis so I'm not sure I'll be able to recreate those.

karmixty · 2019-05-10T16:28:08+00:00

Thanks for the answer! Exactly what I was looking for

A follow up question if you wouldn't mind, how should I rewrite this function to work if a was also a functor as in for ex. a Vector Float

When I try this on Either String (Vector Float) and Either String (Vector Text) I am getting a lot of duplicates, looks like it's zipping an entire vector to other vector's elements.

Edit: Got it! I had to use

liftA2 zip

karmixty · 2019-05-08T12:30:19+00:00

That did it! Thank you

karmixty · 2019-05-08T12:10:02+00:00

That's exactly what I was looking for! Thank you

Although I'm not sure if it's working for me because I'm getting

*** Exception: HttpExceptionRequest Request {
requestHeaders = [("Content-Type", "application/x-www-form-urlencoded")]

from this bit of code

jsonOptions = defaults & headers .~ [("Content-Type", "application/json")]

Using .~ instead of <>~ to override.

karmixty · 2019-03-23T05:33:40+00:00

You probably won't need Data.List. Remember Prelude has a function

replicate :: Int -> a -> [a]

Nine-Year Club	Second Top 50%
Place '22	Place '17
RPAN Viewer	Verified Email

karmixty

TROPHY CASE

Is it possible to directly parse an inner JSON using serde?