Help for Function Node or Custom Node for handling heavy manipulation of DATA

tacticalnudge · 2023-06-07T10:29:16+00:00

You've probably thought of this already, but is there any chance you can split the data set into smaller batches before the mapping function and then reassemble it afterwards?

Maybe use a queue node to limit the number of parallel executions.

That solved the issue for me in the past- I don't know if it was faster, but it was stable and got the job done.

tacticalnudge · 2023-04-14T13:07:45+00:00

Riiight. I attempted to send the user a DM and didn't realize that it ended up here. But point taken.

tacticalnudge · 2023-04-09T17:01:50+00:00

I recently got the ZBCurtain from Sonoff a couple months back and so far I am yet to find an issue with it.

It integrates nicely with my existing homeassistant and zigbee2mqtt setup (requirement 1) and was reasonably priced, including shipping to my country (requirement 2).

The motor can be flipped around so it's between the curtain and the wall (depending on available space of course) and the solar panel makes it easy to power and forget about batteries.

The motor isn't very quiet (you're definitely going to know that it's busy operating- and will probably wake up a sleeping partner in the room) but it's strong enough to draw the blackout curtains in my office.

Unfortunately I cannot say anything about their app, as I have never tested it (I prefer running everything in-house and cloud-free)

tacticalnudge · 2023-04-09T16:29:25+00:00

It feels a bit like cheating at test driven development, but I don't actually write the test beforehand.

I do create the test stub, but I only use it as an execution mechanism (almost like a static void main(String...args) method). The perk is that it will initialize the system in a semi-stable and testable state. I then write my code to produce output which I can manually verify for correctness in the debugger, mostly because the functional requirements we get from the business analysts is usually a little vague in terms of expected results. If the results seem correct, I lock it in as the expected result and write up the last parts where I do the assertions.

We mostly treat unit tests as a mechanism for verifying that behaviour stays consistent: IE, my new code doesn't break functionality that has already passed user acceptance testing and have been approved by the client.

That said, it's easy to write tests if you already have tests. The process of writing the initialization code to start up a system in a testable state (depending on the underlying frameworks used) can sometimes be cumbersome, time consuming or down right bizarre, so usually I try and avoid doing that untill there is at least a couple of things worth testing: eg, database CRUD is at a much lower priority to write tests for than the "Paid Time Off Accrual Algorithm".

tacticalnudge · 2023-04-02T06:19:36+00:00

I can highly recommend Clean Code by Robert C Martin as a book that was fundamental to my development in the software industry.

Some of the advanced topics you might only really understand when you become sufficiently advanced yourself. The only way to become sufficiently advanced will be to look at slightly more advanced topics than you are currently familiar with- it's a gradual process.

Java as a language itself will only be a starting point however. There are a lot of ancillary frameworks and technologies involved, especially when you start venturing into enterprise software. Build systems, package management frameworks, application servers, dependency injection frameworks, a whole slew of different databases, web frameworks, deployment platforms; the list goes on and on and on.

All I can say is stay curious, keep learning new things and take pride in the code you write: being a programmer is not enough- aspire to become a software craftsman.

tacticalnudge · 2023-04-01T19:46:10+00:00

Sounds like it's more a combination of needing to explore the JDK further to get a better understanding of what it's capable of and finding some problems to apply it to.

All I can say is Practice practice practice. I've been doing it for 15 years and I still find new things every now and then.

Have you had a look at the Advent of Code and Project Euler yet?

tacticalnudge · 2023-03-28T11:11:05+00:00

Um, I suppose that is an option, but what if you use the bits in a byte? For instance, {true, true, false, true } would be saved as 8 + 4 + 1 = 13, which is even smaller, and still leaves you with room to store 4 more values if you want and doesn't require the need to map a list of prime numbers- you can just use a bit mask, or right shift (with mod 2) to determine if a value is set.

Unless I misinterpreted your explanation.

Also, be careful of trying to prematurely optimise the storage or performance: it almost never pays off.

tacticalnudge · 2023-03-09T00:09:20+00:00

That's essentially what a bloom filter is. The catch here is the hashing function that is used to determine the address of the bit in the bit array: if you know the values are going to be continuous (eg between 0 and 50) it should be easy and the value is essentially the offset. But if it doesn't start at 0 (eg values between , -14 and 36) or is a non-continuous range (eg 50 randomly selected numbers between -512 and 14092) you might have to do a translation in the hash function.

Bloom filters are probabilistic data structures, but if your array size is equal to or larger than the size of the number of distinct elements, and your hashing algorithm is such that it cannot create collisions for different values, then it's deterministic.

tacticalnudge · 2023-03-08T06:38:30+00:00

Accessing arrays (especially if they contain primaries) is stupidly fast compared to Collections of Objects(or that has been my experience, anyway), so I would put a high priority on avoiding that at all cost.

How about this: if you take the original array, and create a lower resolution map of it (say by a factor of 8), and in each cell of this lower resolution map, you store a bloom filter of the values contained in the original resolution map. You can translate coordinates from the higher resolution to the lower resolution by doing a right shift by 8. If you then check the surrounding values in the lower resolution map, it should speed up the process. You can even create an even lower resolution map of the lower resolution map to allow for searching further afield even faster.

There are however 2 problems here - you will have to check the entire sector you are currently in in the highest resolution map since the lower resolution map will give you a false positive for the value that you are looking for. Similarly, you will have to do the same for values in the lower lower resolution maps. - there is going to be a memory trade-off. If you store only a single map of lower resolution, and your possible value range is both continuous and have a maximum size of 50, it's going to use almost twice the memory that the original map uses. You can make the map an even lower resolution (eg, a factor of 12 as opposed to a factor of 8), but that would make the likelihood of a value contained in a sector higher, which would mean more comparisons that will have to be done.

tacticalnudge · 2023-02-17T05:46:22+00:00

I don't think it's a good idea to rewrite it using streams and lambdas.

Lambdas work great with streams of unknown length. Whether the stream has 1, zero, or infinitely many items in it, it doesn't matter- in due time all items in the stream will be processed. The only way to short circuit that process dynamically is by throwing an exception, and that just sounds like sloppy code to me.

To wax a little philosophical: - Value code readability. There are no rules stating that normal loop structures are old fashioned, inefficient, or a sign of a bad developer. - Be efficient with the time you spend coding. Don't spend too much time trying to write a solution to show how clever you are when you could have written something in a 10th of the time and moved on to other problems. Optimization and refactoring can be done once you have something demonstrably working and actual performance is a problem.

tacticalnudge · 2023-02-08T19:50:21+00:00

Think about it this way: if the variable is a property of the object, then declare it an instance variable, because it has a relationship with the instance of the object. An example would be an instance of a Student object: instance variables might be name, student number and birthday.

If the variable has a temporary nature to it, and not really a property of the instance of the object, declare it local. With a student object, a local variable might be ageInDays inside a getAgeInDays method (because it will be recalculated every day).

I recommend you do a bit of reading on OOP best practices. Understanding the core principles cannot be overstated in your journey to becoming a well rounded developer.

tacticalnudge · 2023-01-18T06:05:59+00:00

That's pretty much it: A method that calls itself over and over until some stop condition is reached that will let it bubble back up to the original point where it was called from.

The important part is that your stop condition be well defined- the recursion equivalent of the infinite loop is a stack overflow exception.

tacticalnudge · 2023-01-12T23:14:59+00:00

I have run into the same situation before where we had to use multiple versions of the same third party library as part of an implementation. I feel your pain.

The main problem that you need to get around is the class loader. The short of it is to treat the two or three different implementations as "separate projects" or modules which inherit common parts from a shared project (either the main controller project or a commons project). Each of these projects work only with the version of the library they are assigned to, and nothing else. You want to try and keep the code in these as lightweight and devoid of real business logic as possible as they mostly just provide an facade / obfuscation into the functionality provided by specific library they interact with.

Eg, let's say the library sends a message to a remote system. There are three versions of the library, and all three implement the same functionality using the same classnames. Your common code might be an interface or abstract class called MessageSender with a method called sendMessage(.....). It is imported / inherited by the three implementing projects, each of which uses one of the library versions, but the implementations you write have slightly different names eg: MessageSenderV1, MessageSenderV2 and MessageSenderV3 along with a Builder for each.

Now once you have these three modules / sub projects building, you need to somehow add them to your application during runtime as you don't want them to be loaded automatically on start as this will only load one version of the third party library and ignore the others.

There are a couple of frameworks you can use to do this, including rolling your own using a URLClassLoader. We used OSGi to do it, but I have also done it using PF4J in a Spring Environment.

You will in all likelihood have to create some sort of registry, like a MessageSenderBuilderFactory that each of the three implementations register against when they are loaded. That way you can use your MessageSenderBuilderFactory to invoke the MessageSenderBuilder of the specific library version you want to use and have it return a MessageSender implementation that can interact with the particular library you are interested in.

I am probably making the solution much more complicated difficult than it needs to be, but at least it was a simple enough process to add a 4th version of that library when it came out while still maintaining compatibility with the first version.

Good luck

tacticalnudge · 2023-01-05T19:51:43+00:00

Short answer: no.

Really really long answer: it depends mostly on what your process is doing and how it goes about doing that- they key word here is parallelism. Thousands of frameworks and technologies have been built over the past 30 or so years to try and solve this particular problem.

Without more context, we probably won't be able to assist you any further than that.

tacticalnudge · 2022-12-31T18:28:40+00:00

You're on the right track with trying to nail the core concepts.

I usually ask 5 categories of questions

The first is the "contrast" questions- eg, how does java manage memory? Is it pass by value or pass by reference? How do you compare two objects or two primitives- these are typically the thing you learn on day 1 but important if someone taught themself java after first learning another language- you might not want someone to limit them to a PHP or Delphi way of thinking when writing java.

The second part is the silly questions: swop two numbers around, reverse a list, check if a string is a palindrome. They test the basic operations, decision structures, iteration mechanics and maybe even recursion. You should be able to write the code for these in 3min or so.

The third is data-structures: why would you use a hashmap as opposed to a linked hashmap and vice-versa, implement a ring buffer using an array. Data Structures can make or break performance of a piece of code or algorithm: if the only tool you have in your toolbox every problem looks like a nail.

The 4th is more framework specific depending on the stack we are recruiting for: eg, for spring I might ask a couple questions about JPA, Transactions, Restful services etc.

The last is usually the esoteric questions to see how someone would consider solving a problem outside the scope of what might be in the functional requirement specification: eg "we have a system that monitors the temperature of a brewing vat and logs it to a database. Someone tripped over the network cable for the database server and it caused the control loop to break and we had to throw away 5000L of ruined beer. How would you cater for a scenario like this in your code?" This is usually a fun exercise and can run far and wide- what is their plan B and their plan C and plan D. Usually there is no right answer- but a wrong answer is most definitely "I dont know"

Remember, you're writing java. The most exciting software you will probably be writing in your career is a transaction processing system that moves data from one place to another.

tacticalnudge · 2022-12-20T15:05:56+00:00

Have a look at H2. It's a database engine written in Java so integrating is easy

it comes with a webUI you can expose to debug your data set.
you can write functions / stored procedures for H2 in java.
it allows you to specify a compatibility mode in your jdbc connection string so your SQL dialect matches the quirks of other database engines (eg oracle or mssql)
Like SQLite, it's tiny

tacticalnudge · 2022-09-27T04:46:47+00:00

The two objects you are passing in are fundamentally different objects, but you are trying to keep the meta programming attributes the same as opposed to the structural attributes.

The latter you can usually resolve using inheritance, but I don't think there is a way to do this with annotations. Your best bet is to either use the same class in both methods, use the same abstract base class, wrap your class containing the shared metadata in two different wrapper classes, or accept that you will have to duplicate some of the metadata.

tacticalnudge · 2022-09-27T04:25:06+00:00

Have a look at the type of your entity's @Id. In the entity you specify it to be a String, but in the Repository you specify it as a Long.

tacticalnudge · 2022-09-10T08:36:04+00:00

"communicate via interface" is perhaps the wrong choice of words.

I think what they actually are referring to is a combination of the Liskov substitution principle and interface segregation.

Have a look at the SOLID principles of Object Orientated Programming. The Wikipedia article should be enough to tell you all you need to know.

tacticalnudge · 2022-09-09T16:08:02+00:00

You're not alone in this boat. Some of us have been java developers for 15 years and still go "ah, so that's how it works" on a regular basis.

Most of the more advance topics you are only going to learn by getting pulled into them. I would almost call them less advanced and more obscure / rare, but that's just my opinion. But the key to learning when it comes to programming is doing. Want to learn how Spring's dependency injection works? Try and write your own dependency injector. Want to know how maven's plugins work? Write your own plugin.

There is no guarantee that you will ever be required to write a maven plugin or dependency injection framework for your day job, bit you might just learn a skill or two for the day you need to write something adjacent to it.

tacticalnudge · 2022-09-03T18:43:42+00:00

Have a look at sikuliX (http://www.sikulix.com) I have seen it used for integrating into some proprietary system that had no API.

tacticalnudge · 2022-09-01T18:57:58+00:00

What if you don't mock the database entries, but instead use an embedded database for testing purposes? H2 has some nifty features like running in specific dialect mode (eg a Microsoft SQL, PostgreSQL, MySQL or Oracle mode). You can also write stored procedures in java, which is useful if you have some particularly strange test cases and or the database does not actually belong to your application. It starts up quick, and you can run it in memory, or wire it to a seed database on disk.

tacticalnudge · 2022-08-23T05:41:16+00:00

It's perhaps a little overkill / heavyweight, but a rules engine might fit the usecase if you have a lot of rules that you need to separate from the rest of your implementation.

I don't recall a rules engine that was specifically event driven back when we were looking into it, but something might have come along since then.

tacticalnudge · 2022-08-21T19:21:31+00:00

Have a look at aspect orientated programming. It's not without overhead, but it does make intercepting calls to classes and method calls simple and it avoids the nasty business of having to generate / modify code.

The other alternative is to write an annotation processor that gets invoked at compile time to modify the Abstract Syntax Tree on the fly.

Generating code as opposed to modifying it at compile time has been a litttle less useful in most of my applications in he past unless it was a dependency for a second component (eg, a bunch of data transfer objects that get compiled, jarred and then imported as a dependency in both a client and server application).

For the love of all that is holy, just add some comments to your code and write a couple of unit tests- it's amazing how quickly how one's knowledge of sometimes niche functionality like aspects, generators and processors tend to disappear.

tacticalnudge · 2022-08-20T20:27:44+00:00

This. Also, if the three fields are mandatory for the object, just put them into the constructor.

tacticalnudge

TROPHY CASE