. industry tracker « Torchwood Archives
Torchwood Archives
21Oct/13Off

LMeve dev blog #2

A few months ago I've written a post about how the development of LMeve goes, and I feel it's in order to write another one, because a lot of things have been changed and/or improved since that last post in May.

New Features

In the last post I have outlined what new features we want developed and why. Due to time restrictions I haven't been able to implement everything from that list, but I have covered quite a sizeable amount.

lmeve-materials-drake First off, the complete materials list and production cost estimation. Every manufacturer in EVE wants to know which items have the best profit margin. Of course prices of resources fluctuate, so the production and pricing strategy has to be revised in regular intervals.

The material data comes from the Static Data Dump and is correctly affected by the ME level set in the fields seen above (in real time!). If user has the correct permissions, it is also possible to save the ME and PE in the database for future reference (why save it? for automatic kit building of course). Prices are being downloaded from eve-central.com using their own API.

lmeve-kitsSecondly, after one of our members put all the ME levels from our BPOs to the app (thanks!) it was possible to implement the Kit Builder. What is it? As you remember, every member has a list of tasks, that have to be completed within a given month. Each member has a bunch of labs assigned to them at one of our POSs, and has to refill them with materials to maintain the production lines busy. Previously everyone had an excel file with the amounts of materials needed to complete tasks for a given month. Excel. File. Why do we have to use Excel, if we already build tools for thisk kind of reference?

Each task has two buttons now: Full kit and Remainder kit respectively. What do they do? If you press the Full kit button, our app will display a list of materials required to make all the jobs under the selected task (enough for an entire month). The "Remainder kit" button will display the amount of materials required to finish the task (entire month minus what has already been done).

lmeve-tasks1Speaking of Tasks, another one of our members requested a feature that shows the jobs currently in progress (and when they finish). This let's members know when they can log on to cycle their jobs. I think I might want to make two views of it, one summary view (aggregated by typeID and only display the time when the last item of that batch is finished). The current "complete" list of jobs in progress would become the second view.

lmeve-database-drakeNow something else entirely. Ever heard of CCP WebGL? Well, it's a javascript framework that allows displaying 3D ships and other game assets on a webpage. It's the same tech that can be seen in the Ship Viewer on the eveonline.com page. Well, I have incorporated it to our Database page. So when you're away from EVE and would like to spin your ships - feel free to do so. Since the window is rather small, I have decided to give it a "fullscreen" button, for all the ship-spinning pleasure (pressing F11 to first make the browser go full screen makes it look even better).

lmeve-buy-calcIn the previous post I have also mentioned the Buy Calculator. We used to have a gdocs spreadsheet for that. What does it do? Aideron Technologies has a buyback program, which allows corp members to sell minerals, datacores and other materials to corp at Jita buy prices. Saves our corpies going to Jita to sell their goodies, and saves us the trip to buy the materials from Jita. A win-win scenario for everyone. It was only matter of time for the Buy Calc to arrive in LMeve as a web form. It's worthe mentioning, that it behaves just like the gdocs did, only better. See, spreadsheets have this fun feature of selecting next cell when you press enter. It makes entering a lot of data quicker. Web forms don't do this by default, so I had to write some javascript to make it behave like spreadsheets do. What gdocs was unable to do, was displaying a summary window that tells you exactly what to put it the in-game contract. Moreover, we save the pricings in the database, and then match it with in-game contracts. This way we don't even have to manually verify all the prices in game - everything is done automatically! And it works in IGB, too.

Speaking of Market features, we now have a list of Market Orders, too. First, to see what is the best-seller item, secondly, to see which items have been already sold out and need to be produced to satisfy the demand. Unfortunately our market information is classified and there is no screen of this feature.

lmeve-timesheetPreviously I've mentioned that our corp members get paid for their hard work. Well, they do! But some players have more than one character involved in production, and would like to see how much they've earned in total on all their characters. A very nice feature request, and quite easy one to implement. The Timesheet module now shows the characters of the logged user on top of the table with a nice wrap-up with the sums for all characters owned. Also one of our members have got himself a new nickname: "The King of Alts".

lmeve-graphs-2-apiHave you noticed the graph by the way? Well, a few Graphs have arrived in LMeve, too. First the one with corp production activity, that shows the number of jobs that have been installed on each day. Next one is the EVE API monitor. It shows how long it took for the poller to refresh all the API feeds. Since some feeds are refreshed less often than others, some poller cycles take longer, because more data needs to be downloaded (and then parsed into the DB). The last graph is the wallet activity, which shows us when goods have been sold (or materials bought).lmeve-graphs-1

The last feature is skins - the app is CSS skinable, and there is a distinct possibility that Aideron members can be very surprised that the screens here look very different to what they are used to. That's because there are three skins two skins! One is very similar to Aideron Robotics (because our corp is a spin-off from AR), second skin, which looks very much like EVE Online forums (CSS isn't a direct copy though, it's only very similar colors). The last one is very simple, and can be considered "work safe".

lmeve-skins3 lmeve-skins1 lmeve-skins2

The future

Whew! That was a productive five months there, not only because LMeve has several new features now, but because of the tool we were able to break our profit, turnover and production records every month!

There is still one big feature that needs to be coded (I didn't have the time to do it yet) the Inventory. We have to track the materials and sub-products (like capital parts or Tech II parts). Inventory will show us how much of given type we have, and how much we need to produce (or buy) to keep the stocks full. Maintaining the smallest stocks, that allow uninterrupted production is one of the crucial logistic tasks for every corp. Every time we run out of materials and members can't cycle their jobs is time when we don't make profits.

The other ideas mentioned in the previous post are still in the queue to be implemented as well:

  • Industry Goals (similar to Tasks, but at the corp level). Ideally I would like to draw pretty production lines for Tech II and Capital items with all the intermediate products.
  • Linking characters to accounts using EVE API (currently admin has to link characters to account manually)
  • Simple character API reader to run background checks on prospective recruits.

PS. The app is in closed beta (Aideron Technologies only). Open Source release date is still to be determined.
PS.2 As of 23rd of Oct the Aideron skin has been removed and will be no longer available. A new skin will be created in it's place.

23May/13Off

On the development front – LMeve dev blog #1

Industry Tracker - Login Screen

When I first started this project, I knew it will take more than a day to complete. In fact, I have assumed it will take at least a month (1-2 hours a day) to finish basic features. I was wrong: it took one and a half month. The result however, is better and more functional than the original Specification required.

First I would like to say a big thank you to my corp mates, who are testing LMeve restlessly. Beta test has also raised some excellent improvements and ideas, some of which are implemented already, and some have been added to the drawingboard. Thank you guys!

Features developed since the last post

Tasks is a module, which allows planning and tracking industry jobs over the course of one month. Currently the only person with edit rights is the CEO, so he can plan manufacturing, invention and copying for each corp member. Members can view their own task list, which shows the completion state of their tasks.

LMeve task tracking page

LMeve task tracking page

It looks almost exactly like the Task page on Marcel's aideronrobotics.com (this was one of the specified requirements). Crysis McNally, one of our members has proposed to add a tooltip over Progress and Success bars, which shows how many jobs have been already done. This way corpies know exactly how many more runs are needed to reach 100%. This is one of the features we've missed in the original aideronrobotics.com industry tool.

Database is a module mainly used for planning and reference (and since there is no "Kit Builder" yet, it also tells the amount of materials required for production). It's also the first module to display price data from eve-central.com (data itself has been downloaded for more than a month). We plan to add a drop-down ME field, which will recalculate required materials, and a setting if we want to get (or stop getting) price data from eve-central.com.

Ah, the beauty of Excel in space ;-)

Ah, the beauty of Excel in space 😉

Wallet has been iterated upon, because previously it would display walletTransactions (market sales) and walletJournal (bounties, contracts, taxes, etc.) separately. Market and contracts are now merged, so we know the actual ISK flow (we are buying minerals from corp members using contracts, and the rest of materials from market).

Tips'n'Tricks for Static Data Dump and EVE API

I have already mentioned it in the first post about LMeve, that EVE API can be tricky at times. So does the Static Data Dump.

As it turns out, telling the correct amount of materials required to build an item is not an easy task, because the data is mixed together: skills and "extra materials" are in the ramtyperequirements table, so you have to tell them apart (hint: join invgroups and use categoryID = 16 to separate skills from materials).

Secondly, the invtypematerials table contains the materials you get when reprocessing the item. It should be the same amount as materials required to build an item, right? Well, it only holds true for Tech 1 items. For Tech II, you have to subtract Tech 1 base item materials from the Tech 2 materials (hint: use recycle column in ramtyperequirements table to find the base item). The weirdest thing however is, that for some items it will give the exact results (for example Improved Cloaking Device II), but for some (like Kronos) you will end up with negative values! Of course the only solution is to hide the negative values from output.

Last but not least, EVE API is having hiccups now and then, throwing 1001: cache is invalid error for wallet transactions, wallet journal and message feeds. The main functionality of LMeve does not depend on these feeds (apart from the Wallet module), but for some people, like Somerset Mahm (and his excellent Blink service) it can be quite a pain. Good luck fixing it @CCP!

Story continues

This is not a formal project, so keeping the scope from growing is not much of an issue (the only budget I have here is time), but there is still plenty of features waiting on the drawing board:

  • Industry Goals (similar to Tasks, but at the corp level). Ideally I would like to draw pretty production lines for Tech II and Capital items with all the intermediate products.
  • Inventory (Material Stocks, Buy Calculator for buying minerals from corp members)
  • Market (Monitoring buy/sell orders, Monitoring market prices for materials and products)
  • Settings (Some values are still not editable without phpMyAdmin right now)
  • EVE API monitoring (to gather API statistics and errors over time, currently I only keep the last value)
  • Linking characters to accounts using EVE API (currently admin has to link characters to account manually)

Want to help us test LMeve?

Would you like to join a friendly group of mature players and get paid for cycling corp manufacturing and invention tasks?

Well, now you have a chance! Apply to Aideron Technologies today! Simply contact our recruiter - Razeu - in game.

PS. Please specify AT in the application form, because we still share forum with Aideron Robotics.

13Apr/13Off

Industry Tracker takes shape

Industry Tracker - Login Screen

In a recent post I have introduced you to the LMeve Industry Tracker. It's a web app, which I'm writing for my corp.

Some time has passed, and despite a hard drive failure, the app is already taking shape. The most important part I have described in the previous post, the API poller, is nearly complete. And it's been working stable for more than a month, which is additional boon. I have decided not to parse all-and-every available api feed (do we seriously need that medal data?), but instead I have chosen to poll the most often used ones. This way I can start working on the GUI, which at this time is the most important thing for us. Of course there is nothing stopping me from adding all other API calls later on.

The following feeds are currently being parsed and loaded to the database:

  • global: ConquerableStationList.xml
  • global: ErrorList.xml
  • global: RefTypes.xml
  • evecentral: eve-central.com/marketstat.xml
  • corp: AccountBalance.xml
  • corp: APIKeyInfo.xml
  • corp: AssetList.xml
  • corp: ContactList.xml
  • corp: ContainerLog.xml
  • corp: CorporationSheet.xml
  • corp: FacWarStats.xml
  • corp: IndustryJobs.xml
  • corp: KillLog.xml
  • corp: MarketOrders.xml
  • corp: MemberTracking.xml
  • corp: StarbaseList.xml
  • corp: WalletJournal.xml
  • corp: WalletTransactions.xml

Each feed is cached according to the API documentation, and if a permanent error is returned, the poller will not attempt to read that feed again.

Me and TheAhmosis (Aideron Technologies' CEO) have decided, which modules have to be up-and-running before we can start using the app for the corp:

  • Timesheet - so we know how much money we owe to our employees (yes, Aideron Technologies is paying their members; wanna become a part of the famous Aideron Industry Program and get paid? Apply now!)
  • Wallet - so we know how much ISK we have made (or lost)
  • Tasks - so every employee knows what to make (and when)

I have added one thing on top of that: role based rights management. My old PHP framework is only using two levels of rights currently: admin and user. In this project however, we would like our corp members to be able to register themselves by using their API keys, so we need a more granular security solution. This way we will also be able to match app username to in-game characters, and display the task list only for characters of that corp member.

Pics, or it didn't happen?

Sure thing:

Timesheet Monthly wallet summaries API Poller statistics Industry overview - by character Industry overview - by item type

29Mar/13Off

New EVE API project – Industry Tracker

Industry Tracker - Login Screen

Industry Tracker - Login Screen

About two weeks ago I have started a new project, which uses XML feeds from EVE API. The new app is supposed to be a new industry tracker for my corp, because the old one is not being maintained anymore. We could probably use DRK Industry Tracker for that, but our Directorship has raised some doubts regarding security, because our API information could potentially be available to a third party. A decision was made to develop an in-house solution.

I have already written an EVE API app in Java to view character data in the past, so I have volunteered to be the chief architect and developer of the new app, thinking that API holds no secrets from me. Of course I was very wrong, but I will get to that later.

An industry planner app will have several users, who need to have access to it. This forces either a client-server approach (which would be an overkill for an app like this) or web based approach. Since browser applications only run when they are being visited, I would need some way to keep API data complete and without any gaps.

That's how the API poller was born

Poller algorithm

Poller algorithm

I needed to write a poller script, which could be run in crontab periodically. This way I could keep the live data complete and up-to-date, without the need for anyone to trigger the API update (by visiting app's web page for example).

EVE API has some best practices, one of them being respecting cache timers (to prevent unnecessary load on API servers). Another one is not to poll a feed that returned a persistent error (for example permission denied). I took these two as the main guidelines for my API poller.

It takes some experimenting

I've mentioned that I was wrong to say that I know the EVE API well. Well, I was. According to the documentation on the official EVE Wiki, WalletTransactions.xml.aspx only takes three parameters:

keyID, vCode, characterID

It means that the actual call should look like this:

https://api.eveonline.com/corp/WalletTransactions.xml.aspx?keyID=12345&vCode=1a23f91d20c&characterID=12341234

That call will succeed. You will not get an error message, but a parseable XML file with no data in it.

After some research it turns out one needs to provide the accountKey, which corresponds to wallet division. Then and only then will the WalletTransactions feed return the actual data. So the correct call will be:

https://api.eveonline.com/corp/WalletTransactions.xml.aspx?keyID=12345&vCode=1a23f91d20c&accountKey=1001

Getting the data is not even half the trick

Industry Tracker - Timesheet

Industry Tracker - Timesheet

The EVE Static Data Dump is about 300 megabytes of data. The live information downloaded by the poller adds even more data to dig through. This means that the most important thing is analyzing, displaying and managing all this data. GUI has therefore several tasks:

  • authenticate users
  • authorize users
  • allow corp members to self-register in a secure manner
  • plan the industry tasks
  • track the industry tasks
  • run reports
  • view the static data dump information
  • enrich live data by connecting it with static data
  • manage settings and API keys

While it sounds simple, the majority of the work will be done here. Each of these bullet points will be broken into even more actions. Working application however, will help our corp make billions, so all the effort is well worth it.