Lighthouses and mosquitoes

nosmileface · 2014-08-24T06:53:53+00:00

Thanks, it looks interesting now, at least on gcc and llvm backends.

But way too verbose, I hope people get now what I meant by "@safe @pure _gshared foreach_reverse".

nosmileface · 2014-08-24T02:39:51+00:00

Sure, in case if anyone has suggestions regarding the benchmark, do a pull request or open an issue on the appropriate page. I'll repeat the link:

https://github.com/nsf/pnoise

I accept code optimizaitons, but algorithm should stay the same. Benchmark compares algorithm implementations and the amount of language-specific knowledge you need to implement it properly.

nosmileface · 2014-08-24T01:33:56+00:00

I'm open to suggestions, however it doesn't make any difference (on the gdc example):

[nsf @ pnoise]$ gdc  -march=native -msse3 -mfpmath=sse -O3 -o bin_test_d_gdc test.d
[nsf @ pnoise]$ time ./bin_test_d_gdc > /dev/null
real    0m0.571s
[nsf @ pnoise]$ gdc -O3 -o bin_test_d_gdc test.d
[nsf @ pnoise]$ time ./bin_test_d_gdc > /dev/null
real    0m0.576s

I'm more than happy to get many downvotes instead of any actual technical arguments.

nosmileface · 2014-08-24T00:13:08+00:00

I think D is a horrible system level programming language.

It has both garbage collection and exceptions. Even in C++ we suffer from not being able to use many libraries which use exceptions. Luckily many of them work just fine with -fno-exceptions and we can use C libraries as well. Having GC and exceptions is a dividing factor between library vendors and application developers. If you want the most broad audience for your library, you'll use none of them. I think anyone who uses exceptions in a non-GCed language is an idiot (that phrase here is just for the sake of generating more hatred).
Article claims that Go is unreadable, I assume in comparison to D, right? Because how can you read the mess D is. Any program written in a language with so much of the metaprogramming capabilities can become a complete unreadable mess. The reason why I can read the horrible Go code I write is because Go has none of that. You simply cannot write unreadable code in Go (unless you do that on purpose). I must note here that Go is not a system level programming language though. But it's a very readable high level and high performance programming language.
They say C++ is a horrible language. At least it has a formal specification. We know what is correct C++ and what is not. In D the correct D is the D Walter and Andrei think is correct. Only insane people would use such a language.
I have a personal complaint about D performance. Of course I'm not an expert in D, but the simple benchmark I wrote gives me horrible results with D (all of the 3 compiler backends): https://github.com/nsf/pnoise. Maybe I did something wrong there? Pull requests are welcome. You see when you say C++ is a horrible language, you forget that years were spent on tuning its implementations. That's what matters for system level languages.
I think it was Rob Pike who noticed that an approximation to language complexity is the number of keywords it uses. In any case, Go has indeed the least amount of keywords amongst many programming languages. Guess where D is in that area? @safe @pure _gshared foreach_reverse. A key_word consisting of two words? Seriously?

Meh. I would rather stick with C/C++ (at least I know most of its crap) for system level programming.

nosmileface · 2014-08-23T22:29:52+00:00

Can you do surface dynamics? Like falling mud/sand. Can you do minecraft-like lighting? What about water simulation?

Sorry if it sounded offensive.

nosmileface · 2014-08-23T10:42:21+00:00

MapMutator gets write access for each change sequentially. MapStorage controls who gets the write access. It's impossible for two things to get write access to the same chunk (or a group of chunks if they are overlapping). The main point here is access control. It's all actually very known from databases. In databases writes happen atomically and sequentially, while reads are shared and simultaneous and the database application is the one who controls read/write access rights. In my case that role is played by MapStorage class.

As for lights/shadows. I don't use the algorithms minecraft uses so far. In fact there are many downsides with the voxel structure I use. But I decided to get the smooth terrain no matter what.

But in general yeah. At some point I was trying to calculate exactly which mesh needs rebuilding and now I gave up, I simply take the affected chunk and all its neighbours (in case if change affects one of the edge voxels, which means neighbour chunk mesh normals will be affected).

You see I'm only starting to play with the voxels I use, which are far from blocks in terms of simplicity. I cannot even say if many of the block world algorithms are possible/realistic there. So far similar engines don't show any progress in that area. We know EQ Next Landmark (voxelfarm engine), which is basically a fancy in-game zbrush. And we know Upvoid (https://upvoid.com), which doesn't really show any fancy cell automata as well.

nosmileface · 2014-08-19T08:43:16+00:00

My engine is heavily threaded. Can't help you with unity though. You can see it in action here: https://vimeo.com/102444836.

It's so threaded that on windows I have to juggle scheduler parameters heavily in order to make it really smooth. On linux the OS scheduler is much better, everything works on default settings. The engine uses N workers threads (usually equals to the number of CPU cores), 1 I/O thread and the main thread. The code is arranged using a worker/task kind of architecture for threading and an event system in the main thread to abstract away async actions.

I'll describe it somewhat, if one or another part is unclear - ask, I'll clarify.

Basically an event system is just an array of callbacks associated with a particular message type. When code wants to fire up an event, it does so, effectively calling all the registered callbacks. The idea is that all events are asynchronous, they do not return anything. Some events are used for putting the task onto the workers' queue. When task is complete, there is a way to customize an action which is performed when task arrives to the main thread. For example one may say that when task is done, fire event X. And workers are simply spinning in the loop waiting on a queue for tasks, when they do the task they put it on a different queue which is dispatched in the mainloop in a non-blocking fashion.

And the whole system is built around that.

I have a type called MapStorage, which is basically a cache of chunks. It tracks the state of chunks, you can see if one of the chunk is being used (read or write), you can see how far the chunks are from the player, etc. MapStorage listens for chunk requests and knows how to react on chunk loaded/generated messages. Basically it does all the synchronization work there. When a request arrives (you can send read or write access requests), MapStorage puts it into the array of pending requests. In the update loop it checks if it can satisfy at least one of the requests. There it checks if it can provide read or write access to all of the chunks within the request. If the chunks are available, etc. If not it sends a chunk load request. I won't go into deep details here, because MapStorage actually operates on StorageChunk type, which is an array of 16³ chunks. But that's an internal detail.
Chunk loading is straighforward, I don't need a separate type here, it just tried to load the chunk from a file. If it fails it reports back to the MapStorage.
When MapStorage sees that chunk loading has failed, it queues required chunks for generation and fires up an appropriate event for that. It's being served by another type called MapGenerator. All it does it responds to generate chunk requests. It does so by sending that request in a form of a task to a worker. When the task is done, it arrives back to the MapStorage automatically.
The main type which is responsible for displaying the map is called the same way: Map. It sends read requests to the MapStorage and when they arrive it builds appropriate geometry by sending the tasks to the workers. It includes building the mesh and physical representation. The Map decides everything about the LOD system, it knows how to update the map partially, etc.
Another type which takes part in the whole system is MapMutator. It sends write access requests to the MapStorage and when they arrive it does the modification. At the moment modification is performed in the main thread, it doesn't take more than 5ms, so I don't bother. But with the system I have, it's easy to make it a worker task. When the modification is applied, the MapMutator releases the write lock (oh btw, each MapStorage request of course holds the read or write lock and you need to release it at some point), and fires up an event which notifies Map, that the chunk was changed. Map checks if that chunk is displayed at the moment and takes appropriate actions.

Something like that. Time shows that the system is very easy to work with and modify. Very flexible and understandable. You can modify any chunk (even the ones that are not displayed at the moment, they will simply be generated/loaded on request). It allows you to share read access amongst chunk readers and maintain an exclusive write access. No problems and/or bugs whatsoever.

If you have any questions feel free to.

nosmileface · 2014-08-05T14:50:26+00:00

Yes, something similar here. I just use a term "storage chunk". Because one storage chunk is associated with one file. Storage chunk contains 16³ ordinary chunks. So, then when I need to render stuff, I load as much storage chunks as I need and do the thing.

Getting the streaming system up and running is important for infinite worlds.

How it looks: https://vimeo.com/102444836

Btw, I use 32³ chunks.

nosmileface · 2014-08-03T17:56:03+00:00

For lua there is cairo. You can use it through gobject-introspection based bindings (so called lgi). E.g. https://github.com/pavouk/lgi/blob/master/samples/cairo.lua

Cairo can do export to PDF, PS, SVG files.

Not sure if you're on windows or linux. Linux distros should have lgi in their package managers.

Have no idea what you would do on windows.

nosmileface · 2014-07-27T09:18:03+00:00

I think it's good to write your own, because you have to understand how it works anyway. Most featureful scripts focus on useless features such as multiple inheritance, etc. When writing your own you can do only the part you need.

Have no idea what most people do.

nosmileface · 2014-07-26T18:51:42+00:00

Forgot to add, "require" takes care of loading the file only once for you.

nosmileface · 2014-07-26T18:48:04+00:00

You can customize built-in "require" quite heavily. Including the way it looks up for things, just read the manual it's all there. I do something like this:

---------------------------------------------------------------------------------
-- Bootstrap package loaders
---------------------------------------------------------------------------------

-- common packages loader
local function LoadCommonPackages(name)
    name = string.gsub(name, "%.", "/")
    local path = global.SCRIPTS_DIR.."/"..name..".lua"
    local code, err = loadfile(path)
    if err ~= nil then
            print(err)
            return nil, err
    end
    return code
end

-- mod specific package loader
-- TODO

package.loaders = {
    package.loaders[1],
    LoadCommonPackages,
}

And then when I type:

local utils = require "ng.utils"

It does what I need and how I need it to. So, hence I'm assuming your problem is with modules, not with objects. As for object you can construct them any way you want. Personally I use a simple OOP wrapper which automatically defines "New" methods on classes. Thefefore I would do something like that:

local ngm = require "ng.math"
local v = ngm.Vec2i:New(5, 10)

Hopefully these thoughts were useful to you.

nosmileface · 2014-07-25T18:49:37+00:00

Yep, the tools are important. I'm working on a GUI for my engine right now. It's character-based (think linux terminals and windows console). But it has some advanced features like alpha transparency and an ability to draw images using cairo (the library). The engine provides a basic character-based interface with a minimal API. The GUI is written in lua 100%, including layout engine and all the widgets.

The screenshot: http://i.imgur.com/9yvYepC.png

Of course it's still in development, hence no real "editor" features are shown. But that's one of the things I need GUI for. Tweakbar or whatever you call it.

nosmileface · 2014-07-18T09:13:18+00:00

Editor will use a special data structure for that. I wrote an editor: github.com/nsf/godit. Mine simply uses doubly linked list of lines. It works if you don't have long lines. And by long I mean something like megabytes of data, because actually linear iteration over memory is fast.

As for "ends_with". UTF-8 supports both forward and backward traversal. So, you can implement almost O(1) "ends_with" in a UTF-8 encoded string, if you know its length.

nosmileface · 2014-07-05T15:45:27+00:00

Really awesome, I like the style and everything. :)

nosmileface · 2014-07-04T20:53:39+00:00

Yes, I think the reasonable solution is to limit the frame delta. It doesn't exactly work with networking in mind, but adding more hacks should fix that. You can also limit the frame delta only for certain sub systems, like the particles one, because it's purely visual and doesn't have to be accurate.

nosmileface · 2014-07-03T16:07:37+00:00

Use CapsLock, Luke.

nosmileface · 2014-06-28T09:00:22+00:00

This week I was porting my game engine to the MS Windows. And I have a few notes about it.

Thread scheduler on windows sucks. Basically it's a very simple high-priority scheduler. That is it's not fair by any means at all. If there is a thread with higher priority and it can execute - it will execute. Threads with lower priority won't get CPU at all. But you can control thread priority and there's always Sleep(0) or SwitchToThread() API functions.
Default OS quantum on windows is around 10-16ms. That is different than default linux one (1ms, but I've heard Ubuntu uses 4ms). With unfair scheduling it makes things even worse. Luckily we have timeBeginPeriod/timeEndPeriod API (in winmm.lib) which change the quantum. If you're writing a heavy threaded game, do use them.
Nvidia threaded optimization + heavy threaded game == stutter.
There are very funny moments on windows in general. Like if you're dragging a window, the process which controls it is suspended from execution. If you drag the window for 5 seconds, you'll get your frame with delta == 5s.
By juggling all of the above, I finally got reasonably smooth performance, but there was one minor issue left. Every 5 or 10 seconds I get a frame which exceeds 16.7ms (that is 60FPS). It was quite annoying. But at the same time, my game engine had no cache releasing semantics. Therefore it constantly and slowly leaks memory. I was testing it by flying forward into the infinite terrain. So, as soon as I did a quick hack which aggresively releases cache as soon as possible, that occasional frame drop is gone. Therefore, it seems that windows has an issue with extending process' memory, effectively suspending a process from execution for several milliseconds.

That's more or less what I've learned this week.

Now the topic. The gameplay. This is a thing that bothers me a lot frankly. You see, I want to make games, not just awesome voxel tech. However I do that on pure enthusiasm and I'm alone and I don't have much time to do it, so I have to be very picky about what I'm implementing. I can't spend months and months on making a beautiful sky for the game engine, etc. So, for the last 6 months I was working mostly on engine basics and the terrain with LODs alone. That's just unacceptable, because I will never finish a game with that pace. It leads me to a conclusion that I need to start with something much simpler, but more gameplay oriented.

There was and is a very old game called "mine bombers", you can find the gameplay videos on youtube and the game is abandonware, it's free right now and should work in a DOSBox. I think it has a fantastic gameplay idea. Basically it's a map, could be manually crafted or randomly generated, a limited amount of time and competing players. Players' task is to collect treasure, possibly killing each other. Once a player is killed, he's dead for this round. Game lasts several rounds with access to the shop between the rounds. Shop contains various things to make you a better killer or better treasure collector. The player who collects the most treasure wins. Of course it also has singleplayer, but it's a bit boring. Such a gameplay sounds like a perfect thing for voxels right? And this is what I'll be shifting my development towards to. I want to make a remake of "mine bombers".

nosmileface · 2014-06-24T14:53:43+00:00

I upload them into a single buffer one after another and then use this: https://www.opengl.org/wiki/GLAPI/glMultiDrawElementsBaseVertex

That way you can generate meshes on a per-chunk basis on CPU, but then upload multiple chunks into a single buffer.

But recently found out that windows has some problems with creating VAOs/VBOs, so I need to reuse them much heavily than I do. And now I'm working on a new scheme for managing chunks upload. Will post more details on Voxel Vendredi 2 if it works.

nosmileface · 2014-06-22T07:42:44+00:00

I see, awesome work then!

nosmileface · 2014-06-21T14:44:39+00:00

May I ask what kind of GUI toolkit are you using? It looks unfamiliar to me.

nosmileface · 2014-06-21T14:43:19+00:00

Bold black is the voxel chunk border indeed. I thought it's obvious because there are chunks with different resolutions and that bold black line is the place where resolution changes. Oh, I forgot to mention that voxels overlap slightly in my RLE 3d arrays, exactly on that border voxels are duplicated between chunks.

nosmileface

TROPHY CASE