Hi #DevFleet! CCP has again published the static data dump before deploying expansion to Tranqulity. I would like to thank @Darkblad_eve here, who sent me and @FuzzySteve a heads up tweet on Twitter last night about new SDE being available. It seems CCP has a new procedure: SDE is deployed on friday immediately preceding the usual tuesday expansion deployment on Tranquility (only database is available for now, icons have not been published yet). Anyway, today I had a few spare mintues to run the conversion. Please find the results here:
The item database has been updated to use the new Static Data dump as well. For example: Nestor stats are up to date now, and Deployable Structures have been move to Structures group, along with other POS and Outpost modules. When you head there, you will find all the new deployables (they will be missing icons until CCP publishes them, though).
This post is entirely for the #DevFleet out there. CCP have outdone themselves by publishing static data dump for a new expansion 4 days before actually deploying it on Tranqulity! It was already available on the Toolkit page last friday.
It means that we – the third party developers – had even more time to convert and adapt the data to our projects. Unfortunately it was not so easy to convert the data this time. As most of you know, CCP has recently switched from SQL Server 2008 to SQL Server 2012, which means the official Static Data Dump has now SQL Server’s 2012 media format. To make the ride even more rough, SQL Server 2012 only runs on Windows Vista and above, and Windows Server 2008 R2 and above. In order to make my development VM compatible with the new Static Data Dump, I had to set up a clean new VM with Windows Server 2008 R2 and the new database version. Fortunately it is possible to do all that legally, because Windows Server 2008 R2 is available as a 180 day trial (which can then be rearmed twice to extend the trial even further), and you can use the Express version of SQL Server without paying a dime.
I have also updated the LMeve Database to use this new dump, feel free to have a look.
Type Icons
CCP has only published a data dump, with icons and renders (probably) coming soon. Don’t worry, I have prepared a “delta” Rubicon icon pack. Simply copy it on top of your existing Odyssey icons:
Icons and renders have now been updated by CCP. Go to the Toolkit page to fetch it.
Feedback
If you find any issues with these dumps, please let me know. This is the first dump from SQL Server 2012, so it is possible some things can be different either in schema or in the data itself. LMeve works just fine, but that’s just me.
It’s one of the changes I wanted to do much earlier on, but didn’t have time to properly prepare LMeve for this. Yes, this is the first part of our project that goes public.
Since the day when I learned that CCP has open-sourced their WebGL library, I wanted to use it right away in our little tool. Even though some time has passed since the tech went public, I have only seen it running on one public website: http://www.eohgames.com/labs Back in summer I had a chat with one of the creators of that site, T’Amber (@PaulOosterman). We have discussed the possibilities of CCP webGL and I have shown him my work on it in LMeve Database module. But to show him this one feature I had to create an account and a new role with just this one module allowed. Lots of work to share it with just one person, eh?
So today I would like to share with all of you the “lite” version of LMeve, which replaces the dirty old Static Data Explorer. It is running the same code that powers our closed-beta Industry app, but it has been stripped down to just have this one module. Say hello to LMeve Item Database!
TL;DR
Check out the new Item Database with 3D ship preview (ship spinning works too):
PS. Some things still need work, for example weapons, munitions and structures do not load properly (there is no code that understands that they are not ships and a different loading method should be used). All ships should display properly though.
When I first started this project, I knew it will take more than a day to complete. In fact, I have assumed it will take at least a month (1-2 hours a day) to finish basic features. I was wrong: it took one and a half month. The result however, is better and more functional than the original Specification required.
First I would like to say a big thank you to my corp mates, who are testing LMeve restlessly. Beta test has also raised some excellent improvements and ideas, some of which are implemented already, and some have been added to the drawingboard. Thank you guys!
Features developed since the last post
Tasks is a module, which allows planning and tracking industry jobs over the course of one month. Currently the only person with edit rights is the CEO, so he can plan manufacturing, invention and copying for each corp member. Members can view their own task list, which shows the completion state of their tasks.
LMeve task tracking page
It looks almost exactly like the Task page on Marcel’s aideronrobotics.com (this was one of the specified requirements). Crysis McNally, one of our members has proposed to add a tooltip over Progress and Success bars, which shows how many jobs have been already done. This way corpies know exactly how many more runs are needed to reach 100%. This is one of the features we’ve missed in the original aideronrobotics.com industry tool.
Database is a module mainly used for planning and reference (and since there is no “Kit Builder” yet, it also tells the amount of materials required for production). It’s also the first module to display price data from eve-central.com (data itself has been downloaded for more than a month). We plan to add a drop-down ME field, which will recalculate required materials, and a setting if we want to get (or stop getting) price data from eve-central.com.
Ah, the beauty of Excel in space 😉
Wallet has been iterated upon, because previously it would display walletTransactions (market sales) and walletJournal (bounties, contracts, taxes, etc.) separately. Market and contracts are now merged, so we know the actual ISK flow (we are buying minerals from corp members using contracts, and the rest of materials from market).
Tips’n’Tricks for Static Data Dump and EVE API
I have already mentioned it in the first post about LMeve, that EVE API can be tricky at times. So does the Static Data Dump.
As it turns out, telling the correct amount of materials required to build an item is not an easy task, because the data is mixed together: skills and “extra materials” are in the ramtyperequirements table, so you have to tell them apart (hint: join invgroups and use categoryID = 16 to separate skills from materials).
Secondly, the invtypematerials table contains the materials you get when reprocessing the item. It should be the same amount as materials required to build an item, right? Well, it only holds true for Tech 1 items. For Tech II, you have to subtract Tech 1 base item materials from the Tech 2 materials (hint: use recycle column in ramtyperequirements table to find the base item). The weirdest thing however is, that for some items it will give the exact results (for example Improved Cloaking Device II), but for some (like Kronos) you will end up with negative values! Of course the only solution is to hide the negative values from output.
Last but not least, EVE API is having hiccups now and then, throwing 1001: cache is invalid error for wallet transactions, wallet journal and message feeds. The main functionality of LMeve does not depend on these feeds (apart from the Wallet module), but for some people, like Somerset Mahm (and his excellent Blink service) it can be quite a pain. Good luck fixing it @CCP!
Story continues
This is not a formal project, so keeping the scope from growing is not much of an issue (the only budget I have here is time), but there is still plenty of features waiting on the drawing board:
Industry Goals (similar to Tasks, but at the corp level). Ideally I would like to draw pretty production lines for Tech II and Capital items with all the intermediate products.
Inventory (Material Stocks, Buy Calculator for buying minerals from corp members)
Market (Monitoring buy/sell orders, Monitoring market prices for materials and products)
Settings (Some values are still not editable without phpMyAdmin right now)
EVE API monitoring (to gather API statistics and errors over time, currently I only keep the last value)
Linking characters to accounts using EVE API (currently admin has to link characters to account manually)
Want to help us test LMeve?
Would you like to join a friendly group of mature players and get paid for cycling corp manufacturing and invention tasks?
Well, now you have a chance! Apply to Aideron Technologies today! Simply contact our recruiter – Razeu – in game.
PS. Please specify AT in the application form, because we still share forum with Aideron Robotics.
In a recent post I have introduced you to the LMeve Industry Tracker. It’s a web app, which I’m writing for my corp.
Some time has passed, and despite a hard drive failure, the app is already taking shape. The most important part I have described in the previous post, the API poller, is nearly complete. And it’s been working stable for more than a month, which is additional boon. I have decided not to parse all-and-every available api feed (do we seriously need that medal data?), but instead I have chosen to poll the most often used ones. This way I can start working on the GUI, which at this time is the most important thing for us. Of course there is nothing stopping me from adding all other API calls later on.
The following feeds are currently being parsed and loaded to the database:
global: ConquerableStationList.xml
global: ErrorList.xml
global: RefTypes.xml
evecentral: eve-central.com/marketstat.xml
corp: AccountBalance.xml
corp: APIKeyInfo.xml
corp: AssetList.xml
corp: ContactList.xml
corp: ContainerLog.xml
corp: CorporationSheet.xml
corp: FacWarStats.xml
corp: IndustryJobs.xml
corp: KillLog.xml
corp: MarketOrders.xml
corp: MemberTracking.xml
corp: StarbaseList.xml
corp: WalletJournal.xml
corp: WalletTransactions.xml
Each feed is cached according to the API documentation, and if a permanent error is returned, the poller will not attempt to read that feed again.
Me and TheAhmosis (Aideron Technologies’ CEO) have decided, which modules have to be up-and-running before we can start using the app for the corp:
Timesheet – so we know how much money we owe to our employees (yes, Aideron Technologies is paying their members; wanna become a part of the famous Aideron Industry Program and get paid? Apply now!)
Wallet – so we know how much ISK we have made (or lost)
Tasks – so every employee knows what to make (and when)
I have added one thing on top of that: role based rights management. My old PHP framework is only using two levels of rights currently: admin and user. In this project however, we would like our corp members to be able to register themselves by using their API keys, so we need a more granular security solution. This way we will also be able to match app username to in-game characters, and display the task list only for characters of that corp member.
About two weeks ago I have started a new project, which uses XML feeds from EVE API. The new app is supposed to be a new industry tracker for my corp, because the old one is not being maintained anymore. We could probably use DRK Industry Tracker for that, but our Directorship has raised some doubts regarding security, because our API information could potentially be available to a third party. A decision was made to develop an in-house solution.
I have already written an EVE API app in Java to view character data in the past, so I have volunteered to be the chief architect and developer of the new app, thinking that API holds no secrets from me. Of course I was very wrong, but I will get to that later.
An industry planner app will have several users, who need to have access to it. This forces either a client-server approach (which would be an overkill for an app like this) or web based approach. Since browser applications only run when they are being visited, I would need some way to keep API data complete and without any gaps.
That’s how the API poller was born
Poller algorithm
I needed to write a poller script, which could be run in crontab periodically. This way I could keep the live data complete and up-to-date, without the need for anyone to trigger the API update (by visiting app’s web page for example).
EVE API has some best practices, one of them being respecting cache timers (to prevent unnecessary load on API servers). Another one is not to poll a feed that returned a persistent error (for example permission denied). I took these two as the main guidelines for my API poller.
It takes some experimenting
I’ve mentioned that I was wrong to say that I know the EVE API well. Well, I was. According to the documentation on the official EVE Wiki, WalletTransactions.xml.aspx only takes three parameters:
keyID, vCode, characterID
It means that the actual call should look like this:
That call will succeed. You will not get an error message, but a parseable XML file with no data in it.
After some research it turns out one needs to provide the accountKey, which corresponds to wallet division. Then and only then will the WalletTransactions feed return the actual data. So the correct call will be:
The EVE Static Data Dump is about 300 megabytes of data. The live information downloaded by the poller adds even more data to dig through. This means that the most important thing is analyzing, displaying and managing all this data. GUI has therefore several tasks:
authenticate users
authorize users
allow corp members to self-register in a secure manner
plan the industry tasks
track the industry tasks
run reports
view the static data dump information
enrich live data by connecting it with static data
manage settings and API keys
While it sounds simple, the majority of the work will be done here. Each of these bullet points will be broken into even more actions. Working application however, will help our corp make billions, so all the effort is well worth it.
The JAVA EVE API Viewerhas just been updated with Inferno 1.2 data dump (so the Iteron Mark IV Amastris Edition shows just fine).
I will also try to provide a data dump conversion for Aura in the near future, so the finest EVE app for Android would finally be up to date with the new skills and items, which have been introduced in Inferno.
If you use Linux or Mac and are looking for an EVE Online character tracker, my App should meet your needs.
I have just uploaded a new version of JAVA EVE API Viewer, which contains some bug fixes and a brand new Inferno 1.0 data dump.
(The Inferno 1.0 static data will become obsolete by this time tomorrow, because CCP will deploy Inferno 1.1. CCP has postponed the rollout of Inferno 1.1. I will update the Static Data as soon as it is available.)
If you use Linux or Mac and are looking for an EVE Online character tracker, my App should meet your needs.
You can choose to download a ZIP archive, or launch the App directly, using JAVA Web Start.
Hi #DevFleet! CCP has again published the static data dump before deploying expansion to Tranqulity. I would like to thank @Darkblad_eve here, who sent me and @FuzzySteve a heads up tweet on Twitter last night about new SDE being available. It seems CCP has a new procedure: SDE is deployed on friday immediately preceding the usual tuesday expansion deployment on Tranquility (only database is available for now, icons have not been published yet). Anyway, today I had a few spare mintues to run the conversion. Please find the results here:
