Crius API changes

industryAs per this dev blog by CCP FoxFour (@regnerBA), industry changes in Crius will also change the way Industry API works.

There will be no new endpoints, but old endpoints will be changed instead. This means old tools will no longer work unless their developers adapt them. But fear not, #devfleet has you covered!

Lockefox (@HLIBindustry) has published tools and sample API data to help other third party devs adapt their apps.

You can find his github repository here: https://github.com/lockefox/CriusDev

  • scripts which handle the new BPO data (to use on this file: <sisi client path>\bin\staticdata\blueprint.db from your Sisi client)
  • sample API outputs from the modified endpoints
    • corp/facilities.xml
    • char/industryJobs.xml
    • char/industryJobsHistory.xml

Great job Lockefox! It’s much appreciated!

PS. I will do my best to add these changes to LMeve before Crius deployment, but due to summer holiday season this might take a few days longer. Stay tuned!

LMeve Installation Guide

Preparation

You will need a host with:

  • access to crontab (so most likely to shell as well).
  • MySQL 5 or above
  • PHP 5 or above
  • about 250MB of free disk space (for icons and EVE Static Data Dump)
  • EVE Online corporation API key(or keys)

First, go to https://github.com/roxlukas/lmeve/releases and download the latest stable version.

The trunk code is unstable and shouldn’t be used in production.

Warning! The guide below is deprecated, see readme.md on LMeve GitHub page for the most recent one!

Setup instructions

There is no installer currently, so there is some setup work to get LMeve to run

1. Initial setup

index.php and the website itself is in ./wwwroot/ directory. If you can set up your webserver root to this directory, please do so.

  • Go to ./config/ directory, copy config-dist.php to config.php and set it up according to your host
  • Set up a random new $LM_SALT value in config.php, then generate admin password hash by using php ./bin/passwd.php
  • copy the password hash to clipboard
  • Download current Types and Icons from EVE Online Toolkit page:http://community.eveonline.com/community/fansites/toolkit/
  • unpack all PNG files from Types to ./wwwroot/ccp_img/
  • unpack all PNG files from Icons to ./wwwroot/ccp_icons/

2. Database setup

  • Import ./data/schema.sql file to MySQL database before using LMeve. Remember to set the db config options in ./config/config.php
  • If in ./data/ there are any delta schemas, import them as well (up to the version you’re installing)
  • Import latest static data dump (can be in a different db schema for clarity, for example you can import lmeve db to lmeve schema and static data in sde_rubicon. LMeve will always use SDE schema set in$LM_EVEDB variable in config.php file) You can download latest static data from Steve Ronuken’s website:https://www.fuzzwork.co.uk/dump/
  • If necessary, change table names to lowercase using script ./data/sde-rename-lowercase.sql
  • Access the database using phpmyadmin or other tool, go to lmusers table
  • Edit record for user admin
  • Paste the password hash from clipboard in the pass field afterwards you will be able to login as admin/admin . Password should be be changed in Settings as soon as possible.
  • Add corp API key (or keys) to cfgapikeys table. keyID goes to keyID, vCode goes to vCode
  • Full corp API key works best, but the app will adjust the amount of visible information according to the rights it has been given.

3. API Poller setup

  • Set up API poller in cron to run every 15 minutes*/15 * * * * [path-to-php]/php [path-to-lmeve]/bin/poller.php

Credits and copyrights

  • LMeve by Lukasz “Lukas Rox” Pozniak
  • LMframework v3 by 2005-2014 Lukasz Pozniak
  • rixxjavix.css skin by Bryan K. “Rixx Javix” Ward

Thanks!

  • TheAhmosis and Razeu – it’s their idea that I had the pleasure to wrap in code
  • Crysis McNally – for excellent ideas and thorough testing
  • Aideron Technologies – for excellent closed beta
  • CCP Games – for making such a great game and providing API for us, developer kind, to tinker with

Rubicon 1.1 Static Data Dump conversion

DevFleetHi #DevFleet! CCP has again published the static data dump before deploying expansion to Tranqulity. I would like to thank @Darkblad_eve here, who sent me and @FuzzySteve a heads up tweet on Twitter last night about new SDE being available. It seems CCP has a new procedure: SDE is deployed on friday immediately preceding the usual tuesday expansion deployment on Tranquility (only database is available for now, icons have not been published yet). Anyway, today I had a few spare mintues to run the conversion. Please find the results here:

You might want to get the new “Ship traits” -> Head to Fuzzy Steve’s page (look for invTraits).

The original MSSQL data dump is available on the official Toolkit page.

LMeve Database updated

The item database has been updated to use the new Static Data dump as well. For example: Nestor stats are up to date now, and Deployable Structures have been move to Structures group, along with other POS and Outpost modules. When you head there, you will find all the new deployables (they will be missing icons until CCP publishes them, though).

Rubicon 1.0 Static Data Dump conversions available

DevFleet

This post is entirely for the #DevFleet out there. CCP have outdone themselves by publishing static data dump for a new expansion 4 days before actually deploying it on Tranqulity! It was already available on the Toolkit page last friday.

It means that we – the third party developers – had even more time to convert and adapt the data to our projects. Unfortunately it was not so easy to convert the data this time. As most of you know, CCP has recently switched from SQL Server 2008 to SQL Server 2012, which means the official Static Data Dump has now SQL Server’s 2012 media format. To make the ride even more rough, SQL Server 2012 only runs on Windows Vista and above, and Windows Server 2008 R2 and above. In order to make my development VM compatible with the new Static Data Dump, I had to set up a clean new VM with Windows Server 2008 R2 and the new database version. Fortunately it is possible to do all that legally, because Windows Server 2008 R2 is available as a 180 day trial (which can then be rearmed twice to extend the trial even further), and you can use the Express version of SQL Server without paying a dime.

Now the results!

SQL Server -> MySQL & SQLite:

YAML -> MySQL:

YAML Certificates schema
YAML Certificates schema

I have also updated the LMeve Database to use this new dump, feel free to have a look.

Type Icons

CCP has only published a data dump, with icons and renders (probably) coming soon. Don’t worry, I have prepared a “delta” Rubicon icon pack. Simply copy it on top of your existing Odyssey icons:

Icons and renders have now been updated by CCP. Go to the Toolkit page to fetch it.

Feedback

If you find any issues with these dumps, please let me know. This is the first dump from SQL Server 2012, so it is possible some things can be different either in schema or in the data itself. LMeve works just fine, but that’s just me.

 

New Item Database on Torchwood Archives (with 3D ship spinning)

It’s one of the changes I wanted to do much earlier on, but didn’t have time to properly prepare LMeve for this. Yes, this is the first part of our project that goes public.

Since the day when I learned that CCP has open-sourced their WebGL library, I wanted to use it right away in our little tool. Even though some time has passed since the tech went public, I have only seen it running on one public website: http://www.eohgames.com/labs Back in summer I had a chat with one of the creators of that site, T’Amber (@PaulOosterman). We have discussed the possibilities of CCP webGL and I have shown him my work on it in LMeve Database module. But to show him this one feature I had to create an account and a new role with just this one module allowed. Lots of work to share it with just one person, eh?

So today I would like to share with all of you the “lite” version of LMeve, which replaces the dirty old Static Data Explorer. It is running the same code that powers our closed-beta Industry app, but it has been stripped down to just have this one module. Say hello to LMeve Item Database!

TL;DR

Check out the new Item Database with 3D ship preview (ship spinning works too):

new-item-database

PS. Some things still need work, for example weapons, munitions and structures do not load properly (there is no code that understands that they are not ships and a different loading method should be used). All ships should display properly though.

LMeve dev blog #2

A few months ago I’ve written a post about how the development of LMeve goes, and I feel it’s in order to write another one, because a lot of things have been changed and/or improved since that last post in May.

New Features

In the last post I have outlined what new features we want developed and why. Due to time restrictions I haven’t been able to implement everything from that list, but I have covered quite a sizeable amount.

lmeve-materials-drake First off, the complete materials list and production cost estimation. Every manufacturer in EVE wants to know which items have the best profit margin. Of course prices of resources fluctuate, so the production and pricing strategy has to be revised in regular intervals.

The material data comes from the Static Data Dump and is correctly affected by the ME level set in the fields seen above (in real time!). If user has the correct permissions, it is also possible to save the ME and PE in the database for future reference (why save it? for automatic kit building of course). Prices are being downloaded from eve-central.com using their own API.

lmeve-kitsSecondly, after one of our members put all the ME levels from our BPOs to the app (thanks!) it was possible to implement the Kit Builder. What is it? As you remember, every member has a list of tasks, that have to be completed within a given month. Each member has a bunch of labs assigned to them at one of our POSs, and has to refill them with materials to maintain the production lines busy. Previously everyone had an excel file with the amounts of materials needed to complete tasks for a given month. Excel. File. Why do we have to use Excel, if we already build tools for thisk kind of reference?

Each task has two buttons now: Full kit and Remainder kit respectively. What do they do? If you press the Full kit button, our app will display a list of materials required to make all the jobs under the selected task (enough for an entire month). The “Remainder kit” button will display the amount of materials required to finish the task (entire month minus what has already been done).

lmeve-tasks1Speaking of Tasks, another one of our members requested a feature that shows the jobs currently in progress (and when they finish). This let’s members know when they can log on to cycle their jobs. I think I might want to make two views of it, one summary view (aggregated by typeID and only display the time when the last item of that batch is finished). The current “complete” list of jobs in progress would become the second view.

lmeve-database-drakeNow something else entirely. Ever heard of CCP WebGL? Well, it’s a javascript framework that allows displaying 3D ships and other game assets on a webpage. It’s the same tech that can be seen in the Ship Viewer on the eveonline.com page. Well, I have incorporated it to our Database page. So when you’re away from EVE and would like to spin your ships – feel free to do so. Since the window is rather small, I have decided to give it a “fullscreen” button, for all the ship-spinning pleasure (pressing F11 to first make the browser go full screen makes it look even better).

lmeve-buy-calcIn the previous post I have also mentioned the Buy Calculator. We used to have a gdocs spreadsheet for that. What does it do? Aideron Technologies has a buyback program, which allows corp members to sell minerals, datacores and other materials to corp at Jita buy prices. Saves our corpies going to Jita to sell their goodies, and saves us the trip to buy the materials from Jita. A win-win scenario for everyone. It was only matter of time for the Buy Calc to arrive in LMeve as a web form. It’s worthe mentioning, that it behaves just like the gdocs did, only better. See, spreadsheets have this fun feature of selecting next cell when you press enter. It makes entering a lot of data quicker. Web forms don’t do this by default, so I had to write some javascript to make it behave like spreadsheets do. What gdocs was unable to do, was displaying a summary window that tells you exactly what to put it the in-game contract. Moreover, we save the pricings in the database, and then match it with in-game contracts. This way we don’t even have to manually verify all the prices in game – everything is done automatically! And it works in IGB, too.

Speaking of Market features, we now have a list of Market Orders, too. First, to see what is the best-seller item, secondly, to see which items have been already sold out and need to be produced to satisfy the demand. Unfortunately our market information is classified and there is no screen of this feature.

lmeve-timesheetPreviously I’ve mentioned that our corp members get paid for their hard work. Well, they do! But some players have more than one character involved in production, and would like to see how much they’ve earned in total on all their characters. A very nice feature request, and quite easy one to implement. The Timesheet module now shows the characters of the logged user on top of the table with a nice wrap-up with the sums for all characters owned. Also one of our members have got himself a new nickname: “The King of Alts”.

lmeve-graphs-2-apiHave you noticed the graph by the way? Well, a few Graphs have arrived in LMeve, too. First the one with corp production activity, that shows the number of jobs that have been installed on each day. Next one is the EVE API monitor. It shows how long it took for the poller to refresh all the API feeds. Since some feeds are refreshed less often than others, some poller cycles take longer, because more data needs to be downloaded (and then parsed into the DB). The last graph is the wallet activity, which shows us when goods have been sold (or materials bought).lmeve-graphs-1

The last feature is skins – the app is CSS skinable, and there is a distinct possibility that Aideron members can be very surprised that the screens here look very different to what they are used to. That’s because there are three skins two skins! One is very similar to Aideron Robotics (because our corp is a spin-off from AR), second skin, which looks very much like EVE Online forums (CSS isn’t a direct copy though, it’s only very similar colors). The last one is very simple, and can be considered “work safe”.

lmeve-skins3 lmeve-skins1 lmeve-skins2

The future

Whew! That was a productive five months there, not only because LMeve has several new features now, but because of the tool we were able to break our profit, turnover and production records every month!

There is still one big feature that needs to be coded (I didn’t have the time to do it yet) the Inventory. We have to track the materials and sub-products (like capital parts or Tech II parts). Inventory will show us how much of given type we have, and how much we need to produce (or buy) to keep the stocks full. Maintaining the smallest stocks, that allow uninterrupted production is one of the crucial logistic tasks for every corp. Every time we run out of materials and members can’t cycle their jobs is time when we don’t make profits.

The other ideas mentioned in the previous post are still in the queue to be implemented as well:

  • Industry Goals (similar to Tasks, but at the corp level). Ideally I would like to draw pretty production lines for Tech II and Capital items with all the intermediate products.
  • Linking characters to accounts using EVE API (currently admin has to link characters to account manually)
  • Simple character API reader to run background checks on prospective recruits.

PS. The app is in closed beta (Aideron Technologies only). Open Source release date is still to be determined.
PS.2 As of 23rd of Oct the Aideron skin has been removed and will be no longer available. A new skin will be created in it’s place.

Project Rorqual

bait-rorqual-burningIt all started back in November 2012. Marcel [Aideron Robotics CEO] asked Aideron Technologies if we can build him a Rorqual. While AR is in FW and not into heavy duty industry, Rorqual can be made into a mobile jump clone station, not to mention it makes a great bait. I have always wanted a Rorqual as well, but since my other capitals have been collecting dust for a long time, I never came down to buying a Rorq. Building it ourselves makes it quite a bit cheaper, so I have decided to finally get one as well.

Aideron Technologies had no Capital blueprints back then: neither for the parts, nor for any capital ships. We had to change that quickly. We have agreed to buy one or two personal capital part BPOs each, and then let the corp use them. Of course corp would research those personal BPOs for free.

Another thing was finding a rather quiet low sec system with factory, where we could build the ship. After some research we found a perfect candidate. Few days of 23,5/7 scouting has confirmed the system is not very active and meets our expectations. Capital Part copies have been made, and in March 2013 we have built our very first Rorqual.

I have started building parts for my Rorqual in May, but because of other tasks (and my absence from EVE) I was unable to finish the ship back then.

2013.05.23.08.53.18

In the background you can notice how LMeve makes it easier to track progress of production tasks 😉 It took a few more days of June to make all required capital parts.

And then came this day

2013.06.19.14.20.19

In about 12 days I will become a proud owner of a Mobile Cloning Station (got modules already prepeared, shy of small gallente ninja-tower and the fuel for it).

PS. Sorry for my absence

The last couple of weeks was a rough ride in RL, to the extent where I was not even able to log on to EVE for several days. I’m still a bit more busy than usual, but you can expect new posts to arrive at my regular rate of 1-2 a week.

On the development front – LMeve dev blog #1

Industry Tracker - Login Screen

When I first started this project, I knew it will take more than a day to complete. In fact, I have assumed it will take at least a month (1-2 hours a day) to finish basic features. I was wrong: it took one and a half month. The result however, is better and more functional than the original Specification required.

First I would like to say a big thank you to my corp mates, who are testing LMeve restlessly. Beta test has also raised some excellent improvements and ideas, some of which are implemented already, and some have been added to the drawingboard. Thank you guys!

Features developed since the last post

Tasks is a module, which allows planning and tracking industry jobs over the course of one month. Currently the only person with edit rights is the CEO, so he can plan manufacturing, invention and copying for each corp member. Members can view their own task list, which shows the completion state of their tasks.

LMeve task tracking page
LMeve task tracking page

It looks almost exactly like the Task page on Marcel’s aideronrobotics.com (this was one of the specified requirements). Crysis McNally, one of our members has proposed to add a tooltip over Progress and Success bars, which shows how many jobs have been already done. This way corpies know exactly how many more runs are needed to reach 100%. This is one of the features we’ve missed in the original aideronrobotics.com industry tool.

Database is a module mainly used for planning and reference (and since there is no “Kit Builder” yet, it also tells the amount of materials required for production). It’s also the first module to display price data from eve-central.com (data itself has been downloaded for more than a month). We plan to add a drop-down ME field, which will recalculate required materials, and a setting if we want to get (or stop getting) price data from eve-central.com.

Ah, the beauty of Excel in space ;-)
Ah, the beauty of Excel in space 😉

Wallet has been iterated upon, because previously it would display walletTransactions (market sales) and walletJournal (bounties, contracts, taxes, etc.) separately. Market and contracts are now merged, so we know the actual ISK flow (we are buying minerals from corp members using contracts, and the rest of materials from market).

Tips’n’Tricks for Static Data Dump and EVE API

I have already mentioned it in the first post about LMeve, that EVE API can be tricky at times. So does the Static Data Dump.

As it turns out, telling the correct amount of materials required to build an item is not an easy task, because the data is mixed together: skills and “extra materials” are in the ramtyperequirements table, so you have to tell them apart (hint: join invgroups and use categoryID = 16 to separate skills from materials).

Secondly, the invtypematerials table contains the materials you get when reprocessing the item. It should be the same amount as materials required to build an item, right? Well, it only holds true for Tech 1 items. For Tech II, you have to subtract Tech 1 base item materials from the Tech 2 materials (hint: use recycle column in ramtyperequirements table to find the base item). The weirdest thing however is, that for some items it will give the exact results (for example Improved Cloaking Device II), but for some (like Kronos) you will end up with negative values! Of course the only solution is to hide the negative values from output.

Last but not least, EVE API is having hiccups now and then, throwing 1001: cache is invalid error for wallet transactions, wallet journal and message feeds. The main functionality of LMeve does not depend on these feeds (apart from the Wallet module), but for some people, like Somerset Mahm (and his excellent Blink service) it can be quite a pain. Good luck fixing it @CCP!

Story continues

This is not a formal project, so keeping the scope from growing is not much of an issue (the only budget I have here is time), but there is still plenty of features waiting on the drawing board:

  • Industry Goals (similar to Tasks, but at the corp level). Ideally I would like to draw pretty production lines for Tech II and Capital items with all the intermediate products.
  • Inventory (Material Stocks, Buy Calculator for buying minerals from corp members)
  • Market (Monitoring buy/sell orders, Monitoring market prices for materials and products)
  • Settings (Some values are still not editable without phpMyAdmin right now)
  • EVE API monitoring (to gather API statistics and errors over time, currently I only keep the last value)
  • Linking characters to accounts using EVE API (currently admin has to link characters to account manually)

Want to help us test LMeve?

Would you like to join a friendly group of mature players and get paid for cycling corp manufacturing and invention tasks?

Well, now you have a chance! Apply to Aideron Technologies today! Simply contact our recruiter – Razeu – in game.

PS. Please specify AT in the application form, because we still share forum with Aideron Robotics.

Industry Tracker takes shape

Industry Tracker - Login Screen

In a recent post I have introduced you to the LMeve Industry Tracker. It’s a web app, which I’m writing for my corp.

Some time has passed, and despite a hard drive failure, the app is already taking shape. The most important part I have described in the previous post, the API poller, is nearly complete. And it’s been working stable for more than a month, which is additional boon. I have decided not to parse all-and-every available api feed (do we seriously need that medal data?), but instead I have chosen to poll the most often used ones. This way I can start working on the GUI, which at this time is the most important thing for us. Of course there is nothing stopping me from adding all other API calls later on.

The following feeds are currently being parsed and loaded to the database:

  • global: ConquerableStationList.xml
  • global: ErrorList.xml
  • global: RefTypes.xml
  • evecentral: eve-central.com/marketstat.xml
  • corp: AccountBalance.xml
  • corp: APIKeyInfo.xml
  • corp: AssetList.xml
  • corp: ContactList.xml
  • corp: ContainerLog.xml
  • corp: CorporationSheet.xml
  • corp: FacWarStats.xml
  • corp: IndustryJobs.xml
  • corp: KillLog.xml
  • corp: MarketOrders.xml
  • corp: MemberTracking.xml
  • corp: StarbaseList.xml
  • corp: WalletJournal.xml
  • corp: WalletTransactions.xml

Each feed is cached according to the API documentation, and if a permanent error is returned, the poller will not attempt to read that feed again.

Me and TheAhmosis (Aideron Technologies’ CEO) have decided, which modules have to be up-and-running before we can start using the app for the corp:

  • Timesheet – so we know how much money we owe to our employees (yes, Aideron Technologies is paying their members; wanna become a part of the famous Aideron Industry Program and get paid? Apply now!)
  • Wallet – so we know how much ISK we have made (or lost)
  • Tasks – so every employee knows what to make (and when)

I have added one thing on top of that: role based rights management. My old PHP framework is only using two levels of rights currently: admin and user. In this project however, we would like our corp members to be able to register themselves by using their API keys, so we need a more granular security solution. This way we will also be able to match app username to in-game characters, and display the task list only for characters of that corp member.

Pics, or it didn’t happen?

Sure thing:

Timesheet Monthly wallet summaries API Poller statistics Industry overview - by character Industry overview - by item type

New EVE API project – Industry Tracker

Industry Tracker - Login Screen
Industry Tracker – Login Screen

About two weeks ago I have started a new project, which uses XML feeds from EVE API. The new app is supposed to be a new industry tracker for my corp, because the old one is not being maintained anymore. We could probably use DRK Industry Tracker for that, but our Directorship has raised some doubts regarding security, because our API information could potentially be available to a third party. A decision was made to develop an in-house solution.

I have already written an EVE API app in Java to view character data in the past, so I have volunteered to be the chief architect and developer of the new app, thinking that API holds no secrets from me. Of course I was very wrong, but I will get to that later.

An industry planner app will have several users, who need to have access to it. This forces either a client-server approach (which would be an overkill for an app like this) or web based approach. Since browser applications only run when they are being visited, I would need some way to keep API data complete and without any gaps.

That’s how the API poller was born

Poller algorithm
Poller algorithm

I needed to write a poller script, which could be run in crontab periodically. This way I could keep the live data complete and up-to-date, without the need for anyone to trigger the API update (by visiting app’s web page for example).

EVE API has some best practices, one of them being respecting cache timers (to prevent unnecessary load on API servers). Another one is not to poll a feed that returned a persistent error (for example permission denied). I took these two as the main guidelines for my API poller.

It takes some experimenting

I’ve mentioned that I was wrong to say that I know the EVE API well. Well, I was. According to the documentation on the official EVE Wiki, WalletTransactions.xml.aspx only takes three parameters:

keyID, vCode, characterID

It means that the actual call should look like this:

https://api.eveonline.com/corp/WalletTransactions.xml.aspx?keyID=12345&vCode=1a23f91d20c&characterID=12341234

That call will succeed. You will not get an error message, but a parseable XML file with no data in it.

After some research it turns out one needs to provide the accountKey, which corresponds to wallet division. Then and only then will the WalletTransactions feed return the actual data. So the correct call will be:

https://api.eveonline.com/corp/WalletTransactions.xml.aspx?keyID=12345&vCode=1a23f91d20c&accountKey=1001

Getting the data is not even half the trick

Industry Tracker - Timesheet
Industry Tracker – Timesheet

The EVE Static Data Dump is about 300 megabytes of data. The live information downloaded by the poller adds even more data to dig through. This means that the most important thing is analyzing, displaying and managing all this data. GUI has therefore several tasks:

  • authenticate users
  • authorize users
  • allow corp members to self-register in a secure manner
  • plan the industry tasks
  • track the industry tasks
  • run reports
  • view the static data dump information
  • enrich live data by connecting it with static data
  • manage settings and API keys

While it sounds simple, the majority of the work will be done here. Each of these bullet points will be broken into even more actions. Working application however, will help our corp make billions, so all the effort is well worth it.