Notifications
Clear all

EOI: Changing Modeling Data format


 Anonymous
Joined: 54 years ago
Posts: 0
Topic starter  

Hi Guys,

A discussion on IRC tonight brought up the topic of our current model data format (which is hand coded in lua, with a basic *.OBJ importer). I feel this is a little clumsy and potentially limits the ability of people who may want to contribute with models, etc to pioneer as the 'scripting' is too cumbersome. It would also be great if we can start to separate the 'content' from the 'engine', but still maintain the flexibility that the current system provides.

An idea I have is to support a new file format (such as Collada (dae/xml)) and use this file to describe the model as well as its functions. We could create an API of pre-defined hooks and settings that when set on the model will 'do stuff'. As a modeler/content builder, you shouldn't have to get down to the technical details of the engine. You just want to create your content and let the engine take care of the technical details.

For example, you could model a complete space station, and add in a few defined 'empty/null' points that define where ads go, or a point where the landing pad is, draw a path through the station with a series of points (a line or a path) and that is the station docking path. A separate or different one can be defined for undocking. The idea is to do as much of the 'content' work in the modeller, and export it out in a format that pioneer will read in and use. Want a flashing light somewhere, then define where on the object, give it a tag and pioneer will draw it. Maybe you want an ad displayed on a particular wall? define the material with a special name (lets call it 'ad') and pioneer can render the add in the space you defined. Maybe you want a ground station with a complex underground hanger/garage layout? Maybe I want my airport to work like an airport.

Ships can be done in a similar way, tag a material for where the ship's id is to be printed.

On planets, different building types can have their own set of tags.. the scope is endless. Waypoints can be set on building modules, allowing for roads to connect from 'this point', etc.

What I am keen to know is other people's opinion of this idea? Especially from those who have built models in the past (I'm looking at you potsmoke), would this make it easier to build and extend models? Would this be too limiting/restrictive?

Yes, this is a rather big task, which is why I want to get everyone's input on it and feedback. Lets discuss it..

Cheers,

Phil.


Quote
s2odan
(@s2odan)
Captain Registered
Joined: 15 years ago
Posts: 1212
 

Providing there is no loss in functionality, them I'm for it.

But IMO the best thing to do is a modification of the model viewer to become more of a modelling tool, to help in the creation of a script. So it could set LOD levels, thruster/decal poistions, sub-models ect. Provided someone doesn't mind writing something like that then I think that would be the best way to go, as it would allow newcommers to use the model-viewer to create their .lua script. And people who don't mind getting their hands dirty can still have some fun with the .lua system.

I feel that one the one hand we have this amazing system for models, which is really only limited by the imagination and creativity of the person making the script, but on the other hand it can be a bore writing the scripts and there's a lot that can go wrong 🙂


ReplyQuote
Marcel
(@marcel)
Captain Registered
Joined: 7 years ago
Posts: 1188
 

I'm not quite sure what I think of this. Part of me is thinking "I'm just beginning to build up some lua skills, and now you want to change it?" the rest of me says, "Yay! this could make things easier!" Your example of the space station is basically what is done now in the script. If things like that could be done visually it could help us math and programming challenged artist types produce better content more quickly. Btw Philbywhizz, this is a bit off topic, but potsmoke suggested that I put the textures I used from his eagles on the space stations in a separate folder, in case the eagles are removed or altered. Since you added a 'stations' folder, may I put them in there, or perhaps stations/textures?


ReplyQuote
ollobrain
(@ollobrain)
Lieutenant Registered
Joined: 13 years ago
Posts: 564
 

My suggestion would be to change it if overall its easier to use. Otherwise stick with the current


ReplyQuote
robn
 robn
(@robn)
Captain Registered
Joined: 13 years ago
Posts: 1035
 
s2odan wrote:
Providing there is no loss in functionality, them I'm for it.

But IMO the best thing to do is a modification of the model viewer to become more of a modelling tool, to help in the creation of a script. So it could set LOD levels, thruster/decal poistions, sub-models ect. Provided someone doesn't mind writing something like that then I think that would be the best way to go, as it would allow newcommers to use the model-viewer to create their .lua script. And people who don't mind getting their hands dirty can still have some fun with the .lua system.

I feel that one the one hand we have this amazing system for models, which is really only limited by the imagination and creativity of the person making the script, but on the other hand it can be a bore writing the scripts and there's a lot that can go wrong 🙂

I have two problems with this approach. One is that its hard to write 3D tools, and they're never going to be as good as something like Blender. Having a model viewer is still useful to make sure that the game does the right thing with your model.

The other problem I have is that the Lua script, while incredibly flexible, is just to easy to screw up. Look at the Panther which is still commented out because it does so much in its dynamic function and destroys the frame rate, but even John couldn't figure out how to convert it to static models. Meanwhile the scripts are impenetrable for most people just wanting to draw a shiny box and drop it into the game.

On possibility would to be to use the Lua script as the interchange format, and have a COLLADA importer write out a script. In that case we still have to deal with the long load times and a lot of complex VM/parser code. If we're going to have an external importer I'd rather that its output was as close to the game internals as possible. If we can guarantee that nobody is writing directly to the internal format then we don't have to retain any kind of backwards-compat and we don't have to worry about people modifying the generated scripts (which also opens the way to models without source, if someone wanted to sell a commercial model pack. I have no interest, others might).

I don't see anything in any of the current models that require the flexibility that the Lua system provides.

Ultimately though the focus is on the user. I want the situation to be that someone wanting to make a model can do all the work they need to within a program like Blender or 3ds Max, export, run the compiler, and drop the output file into the game dir. I don't see why we need more complexity than that, and anything more complex than that just adds overhead.


ReplyQuote
s2odan
(@s2odan)
Captain Registered
Joined: 15 years ago
Posts: 1212
 
Quote:
The other problem I have is that the Lua script, while incredibly flexible, is just to easy to screw up. Look at the Panther which is still commented out because it does so much in its dynamic function and destroys the frame rate, but even John couldn't figure out how to convert it to static models.

Oh yes I remember John mentioning this, I also had planned to fix it but tbh had forgotten 🙂 I'm pretty sure that removing the selector function which would choose between textures will solve that problem, and mean most of the ship can be static as it should be. Ahha, I just found the old .lua which I had saved with the alpha4 package, that should run much quicker... Its most likely in the "official" 😉 alpha5 package if its not in alpha 4.


ReplyQuote
 Anonymous
Joined: 54 years ago
Posts: 0
Topic starter  
Marcel wrote:
Btw Philbywhizz, this is a bit off topic, but potsmoke suggested that I put the textures I used from his eagles on the space stations in a separate folder, in case the eagles are removed or altered. Since you added a 'stations' folder, may I put them in there, or perhaps stations/textures?

Yep, it is far better to keep your model 'self contained' by accessing a texture within its own directory rather than referencing another models (what happens when/if the eagles are moved? your station breaks).

The 'stations' folder is just a placeholder where I was trying out my surface stations - feel free to use it for your own use.


ReplyQuote
 Anonymous
Joined: 54 years ago
Posts: 0
Topic starter  
robn wrote:
Ultimately though the focus is on the user. I want the situation to be that someone wanting to make a model can do all the work they need to within a program like Blender or 3ds Max, export, run the compiler, and drop the output file into the game dir. I don't see why we need more complexity than that, and anything more complex than that just adds overhead.

Would we want to run a compiler after saving the model files? I see that as an unnecessary step. I would rather read in the datafiles directly, and using hooks within the model itself (particular named materials, objects, or even custom properties) that can describe or trigger particular functions (like flash lights, animate a part, place an ad on this material, etc). We define what pioneer expects (and what things are optional) and a modeler can create his/her models to that specification.

I'm still in favour of lua as a scripting language for pioneer 😀 , just using lua to hand build a model and animate it to me is cumbersome, especially when there are far better tools out there to do it.


ReplyQuote
Coolhand
(@coolhand)
Master Chief Registered
Joined: 14 years ago
Posts: 112
 

Something that will convert an animation to Pioneer would be very useful, I could export perfect animations to FFED3D with basically one click while the same thing is apparently tediously complex or even impossible to recreate in pioneer without exporting every frame as a separate model... So lua is apparently restrictive in that sense at least.

I like the idea of a 3d to lua converter, one which will load a standard 3d format, like Collada as suggested perhaps... But the chosen format should ideally support animation channels & multiple material id's.


ReplyQuote
robn
 robn
(@robn)
Captain Registered
Joined: 13 years ago
Posts: 1035
 
Philbywhizz wrote:
robn wrote:
Ultimately though the focus is on the user. I want the situation to be that someone wanting to make a model can do all the work they need to within a program like Blender or 3ds Max, export, run the compiler, and drop the output file into the game dir. I don't see why we need more complexity than that, and anything more complex than that just adds overhead.

Would we want to run a compiler after saving the model files? I see that as an unnecessary step. I would rather read in the datafiles directly, and using hooks within the model itself (particular named materials, objects, or even custom properties) that can describe or trigger particular functions (like flash lights, animate a part, place an ad on this material, etc). We define what pioneer expects (and what things are optional) and a modeler can create his/her models to that specification.

The idea behind a compiler is to not have to perform this step on game startup every time. The current load time is huge for precisely this reason. If the "compile" could be made very fast then I wouldn't bother with a separate compiler, but I'm not sure how fast it can be.

Quote:
I'm still in favour of lua as a scripting language for pioneer 😀 , just using lua to hand build a model and animate it to me is cumbersome, especially when there are far better tools out there to do it.

Oh yes, agree completely - Lua is absolutely the right fit for the general script API. I just don't think it makes sense for models.


ReplyQuote
Luomu
(@luomu)
Master Chief Registered
Joined: 13 years ago
Posts: 131
 

It's cool that we have so many ships and models already but the current scripts can get a bit monstrous.

Anyway, here's some things that could be done:

- Separate ship definitions from model definitions, move them in data/objects/ships/myShip or similar instead of /models

- Write a script for Blender that exports a .lua with all those gun, light positions and other tricky values. Practically use Blender as the ship editor.

- Resource sharing: I would encourage building a core library of textures, materials and submodels and share them between objects. The look will be more unified, we will use fewer resources and in case we add normal (bump) mapping or similar not every model needs to change.

- Nothing we currently have would be thrown away for the time being, if possible

Formats: OBJ is fine, but I couldn't get models from Blender 2.5 to import properly, wrong normals and texture coordinates if I remember right regardless of the export settings. Need to open an issue when I get around to testing it again...


ReplyQuote
fluffyfreak
(@fluffyfreak)
Captain Registered
Joined: 7 years ago
Posts: 1306
 

Is there a description of how the current ship definition scripting works somewhere?

I'm not sure if I understand the way that it works at the moment.

There are some good reasons for switching to a custom format, such as the need to parse the current ".obj" files - with a custom format the model+textures could be stored in a memory/loading optimised format for example. Not sure that would really make much difference but it could open up other pre-processing options(?).

I think the idea of authoring everything in the modeller using named/tagged entities is a good approach. Having skeletal animations created by the modeller/animator could open up some really cool landing gear options alone!


ReplyQuote
fluffyfreak
(@fluffyfreak)
Captain Registered
Joined: 7 years ago
Posts: 1306
 

Nevermind my request for info about the ships, I found a description here: http://pioneerspacesim.net/wiki/index.php?title=3D_Modelling 😀

Looking at the way ships are currently rendered I was quite surprised to find that they often have multiple static meshes, and that lua gets called for every dynamic mesh.

I had thought that lua was just used to set up some initial parameters.

This is quite different to how I'm used to seeing things 🙂

After thinking about it quickly these are the obvious things that could be simple to change in a new format:

    [*:27pxgfrl]All static meshes joined into just 2 meshes - one for opaque geometry (Hull etc) the other for transparent geometry (Windows/Glass)

    [*:27pxgfrl]Textures baked into two sets of textures - reduces number of materials, specular, emission, normals etc could all be controlled via textures if desired

    [*:27pxgfrl]Textures also compressed as DXT1 for opaque, DXT5 for transparent - downside is larger on disk footprint, upside is less memory used ingame, also faster for GPU!

    [*:27pxgfrl]Dynamic meshes placed at locations using 3D Modeller, usage controlled via naming convention "Radar", "Landing Gear 1", "Landing Gear 2" etc

    [*:27pxgfrl]Dynamic meshes would be separate objects with their own named animation ranges, so rate of rotation would be controlled by playing an anim instead of calling script, landing gears would have named conventions for raising/lowering etc.

The first couple of changes could be made with the existing format quite simply. They're mostly changes to data loading and authoring but would speed up rendering. You could apply them to most of the objects actually, I'm just investigating the speed of the city rendering out of curiosity now for example and it seems like there are a lot of models with up to 15 sub stages, not all of those are models but a lot are and I'd bet they could come down to 1 or 2 stages with some care.

What do people think? Is any of this helpful?

Andy


ReplyQuote
fluffyfreak
(@fluffyfreak)
Captain Registered
Joined: 7 years ago
Posts: 1306
 

Just wanted to bump this thread and see if anyone was working on it?


ReplyQuote
laribum
(@laribum)
Crewman Registered
Joined: 13 years ago
Posts: 1
 

How much dynamics do we need on these objects.

Can't it be done in a way that the object definition is done in LUA, compiled into a mem structure including all transformation based animation and then used by the renderer from there instead by calling LUA code to draw and animate the objects?

In the end you would setup the transformation matrix for the object transformation and draw the main meshes as well as all integrated animated meshes using their animation transformation matrices right?

In the current version, it still uses the LUA based approach right?

Best,

Laribum


ReplyQuote