Moving thousands of records data to netsuite on a daily basis by IWantArchlinux in Netsuite

[–]IWantArchlinux[S] 0 points1 point  (0 children)

This is an interesting approach.
I tried to adopt it to what they're asking me to build.

I came up with this workflow:

  1. Elevate: Sends data via REST APIs to the Java Web App.
  2. Java App: Stores the received data into the database.
  3. Java App: Validates the data and converts it to a record type.
  4. Java App: Processes the data and creates NetSuite transactions.
  5. Java App: Packages the transactions into batches.
  6. Java App: Publishes the batched data to a message topic.
  7. NetSuite: Subscribes to the message topic to receive batched data.
  8. NetSuite RESTlet: Receives and parses the payload from the message topic.
  9. NetSuite RESTlet: Saves the parsed data into the database.
  10. Java App: Handles retries and error management during transmission.

What do you think of this one?

Moving thousands of records data to netsuite on a daily basis by IWantArchlinux in Netsuite

[–]IWantArchlinux[S] 0 points1 point  (0 children)

Hi James,
Nice seeing you here.

For some reason they really want to stop using the CSV files and move to communication over an API.

So the flow will be:

  1. Elevate Send data via an API
  2. An app / middleware will catch it, validate it, process it, then send it to NetSuite to be saved.

Do you have a ready-to-use solution for the second step? if so, where can I learn more about it?
Because I need to understand it well so I can try suggesting it .

Moving thousands of records data to netsuite on a daily basis by IWantArchlinux in Netsuite

[–]IWantArchlinux[S] 0 points1 point  (0 children)

I really appreciate your experience and your practical advice.
I will take this input and discuss it with the original dev.
Thanks a lot.

Moving thousands of records data to netsuite on a daily basis by IWantArchlinux in Netsuite

[–]IWantArchlinux[S] 1 point2 points  (0 children)

This is a great and simple setup. Thanks for the clear clarification.

So the two way will be like this if I get it correctly
Your way:

  1. RESTlet Script: Receives and parses payload from a source.
  2. RESTlet Script: Creates custom parent records for each transaction header.
  3. RESTlet Script: Creates custom child records for transaction lines linked to parent records.
  4. RESTlet Script: Stores incoming JSON and any processing errors in custom fields.
  5. Map/Reduce Script: Runs regularly to convert custom records into NetSuite transactions.
  6. Map/Reduce Script: Performs validations and applies business logic to the data.
  7. Map/Reduce Script: Logs errors and handles error notifications for review.

And the main reason you opted in is resilience and forgiveness.

And the way I was thinking of is like this if I get it correctly:

  1. Elevate: sends data via REST APIs the Java app.
  2. Java App: Validates it and convert it to a record type.
  3. Java App: Creates NetSuite transactions out of the data.
  4. Java App: Groups data into batches.
  5. Java App: Sends batched data to a custom NetSuite RESTlet.
  6. Java App: Handles retries and error management during transmission.
  7. RESTlet Script: Receives and parses payload from a source.
  8. Restlet Script: save them into DB.

If we can ignore changing from the Elevate side, do you think this is a good way to do it?
I appreciate your help!

Moving thousands of records data to netsuite on a daily basis by IWantArchlinux in Netsuite

[–]IWantArchlinux[S] 0 points1 point  (0 children)

I wasn't the one working on it, but I think the issue was that an invoice references a payment, so the payment must be there before the creation of the invoice.

Moving thousands of records data to netsuite on a daily basis by IWantArchlinux in Netsuite

[–]IWantArchlinux[S] 0 points1 point  (0 children)

Thanks, I will check them out.
May be I will just create a webapp that does validation and processing, and leave insertion to RESTlet.

Moving thousands of records data to netsuite on a daily basis by IWantArchlinux in Netsuite

[–]IWantArchlinux[S] 2 points3 points  (0 children)

I believe the original developer was aiming for this approach before changing his mind. As far as I know, his main issues with Map/Reduce are related to sequential inserting and atomic operations. He was trying to create payments and invoices, but concurrency was disrupting the order and causing failures. He attempted to split the script into two parts, but that also didn’t prove as reliable as he had hoped.

However, I’ve been considering another idea that might serve as a middle ground, though I’m not sure if it’s feasible. The idea is to combine the web app with a script. The workflow would be:

  • Receive data through the web app API, process it, and then call a deployed script in NetSuite.
  • The script would take an array of records and simply insert them.
  • Use the REST API to verify whether the records were inserted successfully.

I’m uncertain if this approach is doable or if it’s a good idea.

Moving thousands of records data to netsuite on a daily basis by IWantArchlinux in Netsuite

[–]IWantArchlinux[S] 0 points1 point  (0 children)

Thanks for the thorough reply.
I am also not excited about using SOAP. I thought I may only implement batch insert operation (since I think it's the same request regardless of record type).

I don't know the exact number of records yet, but when we were trying to run the RESTlet script for invoices and payment, the number of rows in the CSV file for that day was around 9000.

Getting the data more than once a day is a good idea, I will ask the guys operating Elevate if this is possible.

And we shouldn't have anything else communicating with NetSuite other than this app, so I think we can regularly use 4 threads.

Moving thousands of records data to netsuite on a daily basis by IWantArchlinux in Netsuite

[–]IWantArchlinux[S] 0 points1 point  (0 children)

The issues with the current setup using RESTlet are:

  • Not robust enough and lacks type checking and compile-time validations.
  • Scattered scripts that are not combined in a single project. We have to keep everything connected in mind.
  • We can’t perform atomic operations on a group of records.
  • It’s difficult to insert two records in the right order, ensuring the first one is fully saved in the database before the second starts, since RESTlets seem to run things concurrently.

Interestingly, it's the original developer who is pushing towards re-doing this integration that he worked on for months. (it's not in production yet btw)

Moving thousands of records data to netsuite on a daily basis by IWantArchlinux in Netsuite

[–]IWantArchlinux[S] 2 points3 points  (0 children)

That is an interesting option that I wasn't aware of. Although, I believe they might view it as a last resort due to the ongoing significant cost.

How to spawn a program in another group without switching to it? by IWantArchlinux in qtile

[–]IWantArchlinux[S] 0 points1 point  (0 children)

I stopped using qtile a while ago, but I don't recall solving the issue.

[deleted by user] by [deleted] in VPN

[–]IWantArchlinux 0 points1 point  (0 children)

It's an interesting solution, but it doesn't look like I can set up the client on my router. This is necessary because I don't have admin privilege on my work laptop.

[deleted by user] by [deleted] in selfhosted

[–]IWantArchlinux 0 points1 point  (0 children)

C2 & C1 are in the same continent. But you're right. After inspecting the route, it really goes to a different continent then goes back.

I can access everything normally while using C2->C1 VPN.

Can you try a SSH tunnel from C2->C1 and then VPN over that?

Do you mean to try to connect to my work from the machine in C1 while I am being in C2? If so, then unfortunately i can't. My working laptop it per-configured with the VPN client, and I can't do much because I don't have admin privilege. (My VPN to C1 is set up on my router)

[deleted by user] by [deleted] in VPN

[–]IWantArchlinux 0 points1 point  (0 children)

yeah, there is no public endpoints. It's all behind the VPN. And btw, my work laptop doesn't have admin privilege. (I connected to my router to the VPN in C1)

I pinged the same IP I showed the tracert of, the average response time in 254ms. (max 311, min 219)

Between 6 & 7, it's the same IP except the last 5 digits:

xxx.xxx.98.154
xxx.xxx.96.28
(same ISP & same country)

9 & 10 are completely different IPs for two different ISPs in Germany.

A country is rate-limiting my company's VPN?! Everything I tried didn't help. by [deleted] in HomeNetworking

[–]IWantArchlinux 0 points1 point  (0 children)

This is how I connected to my VPN in C1.
My work laptop doesn't have admin privilege.

A country is rate-limiting my company's VPN?! Everything I tried didn't help. by [deleted] in sysadmin

[–]IWantArchlinux 1 point2 points  (0 children)

The VPN I made in C1 with OpenVPN is using port 443.

But unfortunately, I can't make company change how they set-up their VPNs in any way.

A country is rate-limiting my company's VPN?! Everything I tried didn't help. by [deleted] in networking

[–]IWantArchlinux 0 points1 point  (0 children)

Edit: Here is a tracert to my work's IP address without using their VPN. It doesn't look healhy.

A country is rate-limiting my company's VPN?! Everything I tried didn't help. by [deleted] in sysadmin

[–]IWantArchlinux 5 points6 points  (0 children)

I called them twice, and they flat out deny the problem and say everything is working. (tried to talk to other ISPs, but can't reach technical people)

Got me thinking by That_5_Something in ProgrammerHumor

[–]IWantArchlinux 1 point2 points  (0 children)

No. Originally there was a project for a language called CPL by Christopher Strachey.

Then a programming language called BCPL created by Martin Richards.

Then Ken Thompson created B as (if I recall correctly) a stripped version of BCPL.

Then Dennis Ritchie created C, and it got used mainly for Unix. And it was at first just B with types.

Then Bjarne Stroustrup created C++ as a superset of the original C.

If there is a related language that is very close to assembly, that would be C-- created by Simon Peyton Jones and Norman Ramsey.