LMeve dev-blog: Bugfixes and POS tracking improvements

Hi space friends!

More and more of you have already tried LMeve, which I am incredibly grateful for. This means more people are testing the app and I get more feedback, which is awesome. I also get a lot of questions which I try to answer the best I can and in a timely fashion.

Remember: by trying and using LMeve you help developing it!

Few players also caught some bugs, that I’m happy to announce, have been properly squashed in the 0.1.52 release.

  • Bug fix in db.php “Notice: LM_dbhost is not defined” – added $LM_dbhost to global declaration in db.php
  • Bug fix in Database – “function eregi() is deprecated” – replaced with preg_match()
  • Bug fix in YAML updater – changed ‘DROP TABLE’ to ‘DROP TABLE IF EXISTS’ to prevent errors if the table was not created yet.

One bug was found by my trusty Aideron Technologies beta testers:

  • Non-recurring Tasks (singleton) bug fix – one-time tasks did not display correctly due to a mistake in SQL query

What’s new besides Bugfixes?

  • Poller now fetches StarbaseDetails.xml and CREST industry cost indexes

The first API feed is used by POS tracking code to display accurate and up-to-date fuel levels in LMeve Inventory. The second CREST feed provides NPC cost index to the build cost estimation formula (I currently use an average system index, so you might pay more or less than what LMeve uses for price estimates).

In the next release I will add a “Industry home system” field in Settings, where you will choose your home system. LMeve will then use this system’s cost index in all it’s calculations.

  • New POS management screen – displays fuel amounts and how long it will last
  • There is now a notification when any tower has less than 48 hours of fuel left

new-pos-screen

  • Tech III invention added. You can now add Tech III invention tasks and get price estimates for Tech III
  • New global setting for the default Relic type for use with Tech III Invention (Intact, Malfunctioning or Wrecked)

Download LMeve here

https://github.com/roxlukas/lmeve/

If you are updating from a previous release, please remember to create new tables included in this release using /data/schema-delta-0.1.51-0.1.52.sql

Try-before-you-buy

Well, LMeve is free software, so you don’t have to buy it 🙂 (donations are of course welcome!). But let’s get back to business. Installation requires some technical knowledge, so to make it easier for you guys to try it out, I have prepared a pre-installed Virtual Machine with LMeve, which runs on free VMware Player. You can download the image here: https://pozniak.pl/lmeve-vm/ 

  • it contains LMeve 0.1.52
  • static data is updated to Scylla
  • after booting it shows the IP address to point your web browser to:

lmeve_vm

CCP WebGL updated!

This is probably one of the best news for T’Amber and me in a long time: a CCP dev Filipp (we don’t know his CCP nickname yet) has updated CCP WebGL library to use V4 shaders, and moreover he did a complete model export up to date with EVE Tiamat release. What this means is that third-party projects like LMeve and Jeremy can again display all ships in EVE in full glory with up to date shaders and models!

The new webGL library is a little different from the previous one, because it is using the new SOF mechanism (Space Object Factory) instead of plain .red files.

For example, to load an Onyx heavy dictor, old javascript call looked like this:

var ship = scene.loadShip('res:/dx9/model/ship/caldari/Cruiser/CC2/Kaalakiota/CC2_T2a_Kaalakiota.red', undefined);

and new call is a little different:

var ship = scene.loadShip('cc2_t2a:kaalakiota:caldari', undefined);

You probably noticed that instead of a file name we have three “ship DNA” strings now. The mapping between ship’s typeID and the corresponding SOF entries are stored in graphicIDs.yaml, which is available in Static Data Export:

graphicID description sofFactionName sofHullName sofRaceName
10 Moon NULL NULL NULL
38 Caldari Frigate Bantam caldaribase cf1_t1 caldari
39 Caldari Frigate Condor caldaribase cf2_t1 caldari
40 Caldari Frigate Griffin caldaribase cf4_t1 caldari
41 Caldari Cruiser Osprey caldaribase cc1_t1 caldari
42 Caldari Cruiser Caracal caldaribase cc3_t1 caldari
43 Caldari Battleship Raven caldaribase cb1_t1 caldari
44 Minmatar Frigate Slasher minmatarbase mf1_t1 minmatar

I have already updated LMeve to use the new library, but because of a small issue (more on that below), it currently only works if your LMeve is served using HTTP.

Updated Onyx model

Public LMeve Database has been updated as well. Here’s an example page with the Amarr Confessor (please note you must press “3D” button for the WebGL preview to load).

All roses have thorns

Unfortunately the current version of the CCP webGL library has some drawbacks, too:

  • there are two model/texture sources now (so a simple reverse proxy like the one I’ve developed for LMeve won’t handle it)
  • both sources use HTTP, so modern browsers like Chrome will trigger a security error when trying to load HTTP based resources from HTTPS based website.

The latter is a very simple thing and I’m sure Filipp will be able to correct it. With CORS headers now properly set on CCP CDN we no longer need a proxy to serve models and textures.

New version of Jeremy in the making

With the new webGL library T’Amber sees a lot of potential to improve Jeremy. the new version would be using dynamically generated list of ships (instead of hardcoded list maintained by T’Amber). I will help with all the backend work (for example serving all SOF and invTypes data by JSON API).

Here’s a little teaser:

Stay tuned, because we are working hard to deliver you the best possible out of game ship spinning experience 🙂

AVvDxvj cpj6GjW

LMeve dev blog: important fix in dates, proxy and one more thing

Hi space-friends!

I won’t lie to you, I haven’t been playing much EVE lately. All the spare time that I was able to find was used to improve and fix LMeve. And Aideron Technologies found a big-badda-bug this time.

On the last saturday of January, a corpmate asked me why he can’t see the effect of his fire sales on the corp wallet graph in LMeve. It was the last day of January, and all graphs showed an empty bar for January 31st.

First I thought that poller might have stopped and we don’t get up to date API data. But I was quickly proven wrong, because I could clearly see all the recent market transactions in the database. “Must be a mistake in code that feeds database values to the graph” I thought. But when I checked that code, it only confirmed what I saw on the graph: that the value for the last day is indeed equal zero.

If the data is in the database, then it must be some kind of a presentation error; there is simply no way last day should show zero!

Let’s have a look at the database query used to retrieve the wallet data for a specific month. The clue must be there!

SELECT SUM(awj.amount)/1000000 AS income,date_format(awj.date, '%e') AS day FROM
apiwalletjournal awj
JOIN apireftypes art
ON awj.refTypeID=art.refTypeID
WHERE awj.date BETWEEN '${year}-${month}-01' AND LAST_DAY('${year}-${month}-01')
AND awj.corporationID=${corp['corporationID']}
AND awj.refTypeID <> 37
AND awj.amount > 0
GROUP BY date_format(awj.date, '%e')
ORDER BY date_format(awj.date, '%e');

Everything looks good from here, we SUM all transactions with positive “amount” field, and group them by the “day”. Of course I want output in millions of ISK, so I divide it by 1 000 000. Then we only select records between ‘$year-$month-01’ and the last day of the same month. Everything else is just filtering the correct corporation (because one instance of LMeve can monitor many corps at a time), filtering out player donations (refTypeID 37) and ordering the output by day, so we get a list of values for an entire month, from 1st day till the last.

How come this query will always return zero for the last day? Maybe that LAST_DAY() function actually returns last-but-one day?

SELECT LAST_DAY('2015-01-01');
2015-01-31
one row selected

Hmmm, it works exactly as advertized…

Or does it? Records in the wallet journal table don’t just have a DATE of the transaction, they have a full DATETIME! And what LAST_DAY() returns, is midnight on the last day of month! The entire timespan from midnight of that day until 23:59 that day is… well… filtered out. That explains why the last day was always missing.

How to fix it then?

Well, since we’re missing exactly 24 hours, it’s as simple as adding 1 day interval to the output of LAST_DAY():

WHERE awj.date BETWEEN '${year}-${month}-01' AND DATE_ADD(LAST_DAY('${year}-${month}-01'), INTERVAL 1 day)

LMeve CDN proxy

Now something else. As you know, I am now hosting T’Ambers ship painting tool – caldariprimeponyclub.com. In short, the problem was with the CCP CDN not setting Access-Control-Allow-Origin * in the header for users in AU and NZ regions (or some transparent proxy on the way stripped that header). Regardless, the end result was that T’Amber’s tool was unable to fetch 3D models and textures and ended up with a blank screen. I had a similar issue in LMeve a while back, and I made a very simple proxy in PHP, which would fetch the file from CCP CDN using curl, add the necessary CORS headers, and then send it back to the user.

The original proxy was really crude and could cause unnecessary network traffic, because every time user asked for file ‘A.png’ I would download file ‘A.png’ from CCP CDN, even if I have downloaded this file five seconds ago! Couldn’t I just save the file on disk, so I only retrieve each file once? Well, I’ve added the necessary logic, and the file is now saved in a cache, so I only retrieve each and every file once.

Now the second thing we needed was Analytics. It is a simple thing really: it’s statistics and visibility. I keep a log that contains information about the file (filename, path and its size in bytes) if the file was served from my local cache or retrieved from CCP servers, when it was retrieved, and the IP address of the client (which makes pretty much a standard proxy log). Once I have all this information, I can plot pretty graphs showing how many unique users were there on any given day, how many files they retrieved, and how much traffic they created:

lmeve-cdn-stats-solarized-light

Happy ship painting!

quafe-domi

One more thing

As you have noticed on that last screenshot, LMeve has a brand new skin. That skin is not just a random beige palette. I recently learned of a color scheme which improves readability in text and programming editors, dubbed “Solarized“. That colour set contains 8 monotones and 8 accent colours which have several unique properties. First, they cause much less stress on the eyes, and second, they are displayed correctly on old and new displays, even on intentionally miscalibrated ones.

These two new skins have been added under Settings -> Preferences: solarized-light.css (above) and solarized-dark.css (best for night owls). They are way more minimalistic than original LMeve skins, and they don’t contain too many images, which makes them “work-safe”.

lmeve-cdn-stats-solarized-dark

Hint: you can go further and remove all the images from LMeve. To do that, simply make a copy of your favourite skin .css file (they are in /wwwroot/css/ directory) and add this code:

img {
visibility: hidden;
}

This will hide all the images (character portraits, item icons and so on), but will retain the space that these images would normally occupy. This way LMeve remains completely usable, but becomes a “text only” application, that looks like everything but game 😉

LMeve dev blog: POCO details page and client list!

In November 2013 CCP has allowed high sec NPC Customs Offices owned by Interbus to be destroyed and then replaced with player owned ones. Previously only low-sec Customs Offices were vulnerable. Of course having POCOs makes a very nice passive source of ISK (you don’t have to do anything, apart from an occassional call to arms to defend them). But have you ever wondered how much you earn through your POCO network or who is using them?

In a previous update of LMeve I have added the ability to visualize which POCOs earn the most ISK, and in this iteration I continue the trend to show even more details. You can now choose a specific POCO and see a Details page for it. This page shows how much this specific Customs Office earned in the last 30 days – counting between 30 days ago and today, income in the last calendar month, and in the current one, so you can see trends).

What’s another new thing is the list of your customers. Corp Wallet Journal entries for planetary taxes show the client’s name and planets ID, so by running an aggregate query on that you can see:

  • who is using a specific POCO
  • how often they use it
  • how much they pay in taxes (so you can approach them and offer a discount in exchange for let’s say blue status)

Note: due to this information being obtained from corp Wallet Journal, it won’t show characters who have 0% tax rate, as these people will only pay the NPC tax, and no taxes to your corp at all (so no records are added to corporation Wallet Journal).

PI new planet detail screen

If you have ideas for more features in LMeve, please feel free to contact me. The best way is to raise an Issue on project’s Github page: https://github.com/roxlukas/lmeve. Choose an appropriate category (bug fix, enhacement or question) and write down what you need, or what needs fixing. I can’t guarantee that all requests will result in that feature being added to LMeve, but if the feature looks useful, won’t take forever to code and most importantly, the information required to make it is in the EVE API, it has a high chance of being included in one of the future releases.

Tweetfleet Slack

Just one more thing. Ashterothi, ex-member of Aideron Robotics and well-known host of podcasts such as High Drag and Hydrostatic Podcast, has recently created a Slack team for Tweetfleet. Slack is generally a communications platform aimed at companies, but it has a very nice real time chat (plus a dedicated app for iOS & Android). It is very useful, and a dedicated #devfleet channel also exists, so feel free to drop by and chat with fellow players, CCP devs and third party devs. It’s really fun!

To get invited, simply head to https://www.aideronrobotics.com/tweetfleet/ and fill in the form.

Tweetfleet Slack itself is here: https://tweetfleet.slack.com/

LMeve dev blog: Big POCO update coming

Just as promised in the previous LMeve dev blog, I have recently added the ability to track income from individual planets. Moreover, with the new percentage bars, you will know what are your best planets simply by looking at the table. To check how much a specific planet earned you in the last 30 days, simply hover your mouse above that planet’s name to see a tooltip with the exact number.

lmeve-poco-update

But this is not the only thing added in this release. Some pages require crunching a lot of data, which means they render rather slowly. Most of this data doesn’t change very often, so it seems to be a good idea to generate them once, and then save a cached copy. Then every time users come back to this page, a cached version is displayed, saving everyones time.

One of the pages, the Profit Chart can have several hundred records, which need their manufacturing and invention prices calculated, and it can load for as long as few minutes (depending on individual configuration of course). I used a different approach here: a lazy loader. Instead of one big 2 minute request, I display the page immediately, and as user is scrolling, more data is being loaded into the table in packs of 10 rows.

How to link POCOs with planets & income?

It’s not trivial, but far from impossible. Let’s have a look at columns returned by the /corp/CustomsOffices.xml.aspx API:

<row itemID="101296791423" solarSystemID="30004295" solarSystemName="Keba" reinforceHour="19" allowAlliance="True" allowStandings="True" standingLevel="-10" taxRateAlliance="0" taxRateCorp="0" taxRateStandingHigh="0" taxRateStandingGood="0.05" taxRateStandingNeutral="0.07" taxRateStandingBad="0.1" taxRateStandingHorrible="0.15" />

There is POCO itemID, which stays the same in all API endpoints, there is solarSystemID and system name. And of course the tax settings. But there is nothing about planet itemID here. Maybe /corp/Locations.xml.aspx endpoint will reveal a bit more?

<row itemID="101296791423" itemName="Customs Office (Keba I)" x="58598019193.2515" y="-5937030740.62145" z="-3890977687.95432" />

Okay, so now we have x, y, z coordinates, and POCOs name. In the first version of POCO management screen I simply cut planet name out of this using a regular expression. But to link it with other data we still need the planet itemID. How do we find it? Well, POCO is orbiting a planet, right? And we have cooridnates of all planets in the Static Data Export database – in the mapDenormalize table. What are we waiting for?

CREATE FUNCTION `findNearest`(`x1` DOUBLE, `y1` DOUBLE, `z1` DOUBLE, `solarSystemID1` INT) RETURNS int(11)
 READS SQL DATA
 RETURN (
 SELECT a.itemID FROM
 (SELECT SQRT(POW(x1-x,2)+POW(y1-y,2)+POW(z1-z,2)) AS dist,itemID
 FROM mapDenormalize
 WHERE `solarSystemID`=solarSystemID1
 ORDER BY dist ASC
 LIMIT 1) a
 )

40271955

Bingo! The function returns the closest solar body, in this case, itemID of the planet. We just have to JOIN two other tables from SDE: invNames to convert the planet ID we just got into planet name, Keba I, and invItems to find it’s typeID (Temperate, Barren, Oceanic, etc.).

Ok, now on to income for this specific POCO. For this, we will need  /corp/WalletJournal.xml.aspx endpoint, which contains all the transactions of a corp. The records we are looking for will have refTypeID equal to 96 and 97 (import/export taxes) and the planet itemID will be in argID1 field. Once all joins are in place, we simply have to filter last thirty days and SUM all the amount values.

As usual, the current LMeve version can be downloaded from https://github.com/roxlukas/lmeve

LMeve dev blog: Catching up with Phoebe. Bonus: setting up SSO

Good news! Last monday I had a day off, so I finally caught up with all outstanding tasks, including making LMeve Phoebe-compatible. In the meantime, I started tinkering with the following features:

Vagrantfile, bootstrap.sh

These two files make it easier to set up a host for any app. Trent Bartlem suggested to me on Github to include these files back in August. While the files themselves are pretty straightforward to prepare, I have to install LMeve from scratch somewhere to make sure I included all the necessary packages required to run LMeve. It’s on my to do list, and it should happen sooner than soon™.

Page Caching

Another idea I have is to make the longest loading pages cached (and maybe – just maybe – refreshed in the background, for example by the poller). Code to make this kind of caching possible is already written, and two sample pages (Inventory, Profit Chart) have been added as well. I use AJAX call to request the cached page, and if  cached version is not available (or has expired) I display a “Loading…” sign and prepare a refreshed version of the page. I was thinking about adding a “Force refresh” button, so users can force LMeve to refresh a cached page.

I still evaluate how this improves performance, but it looks pretty good and should become public soon.

PoCo module

lmeve-pocoLast one on the To Do list is expanding PoCo module. It has been recently revamped with features such as displaying last month’s income and current month income prediction. Judging by the talk I had with Dracoth Simertet from RvB (they have a huge PoCo network) and CCP FoxFour, there is a demand for a tool for managing PoCo networks. But what features would you need to monitor your PoCos?

  • Last month income per PoCo (currently only a sum for entire corp is displayed)
  • Planet type the PoCo is orbiting
  • Number of interactions
  • Client list per PoCo (by corp, by char)
  • Reinforced status
  • Do you have more ideas? Please leave a comment if you do!
    • We are only limited with what’s available in the API

SSO integration

I have mentioned a few times already that LMeve can benefit from EVE SSO, but some players were not sure how to set it up. Don’t worry! I’ve got you covered! Here’s a quick guide.

  • Secondly, click “Create new application”
    developers-02
    developers-03
  • Fill in the form. Application Name can be for example “LMeve – <your corporation name here>”
  • Now make sure to fill in Callback URL field correctly. If your LMeve address is https://myhostname.com/lmeve/, then the correct callback will be https://myhostname.com/lmeve/ssologin.php
  • Save the changes.

Once the application has been created, open it again by clicking the “View application” button, and write down these two values:

  • Client ID
  • Secret Key

developers-04

they will be required to complete setup on the LMeve side.

  • Log in to your web server host, open up config/config.php in your LMeve directory.
  • Find these variables, enter your callback URL and the values written down in previous step:
//use EVE SSO - see https://wiki.eveonline.com/en/wiki/EVE_SSO_Documentation
$SSOENABLED=TRUE; 
$SSO_REDIRECT_URL='https://myhostname.com/ssologin.php';
$SSO_CLIENT_ID='sso_client_id';
$SSO_CLIENT_SECRET='sso_client_secret';
//Auth server can be either login.eveonline.com for Tranquility, or sisilogin.testeveonline.com when trying to use Sisi.
$SSO_AUTH_SERVER='login.testeveonline.com';
  • Make sure characters are connected to LMeve usernames. This is crucial for the SSO to work!
    • Go to “Characters” module in LMeve and check if your characters are there.

lmeve-sso

Now you are ready to use EVE SSO to login to your LMeve instance!

LMeve dev blog: Phoebe support delayed

I have a very quick announcement today. CCP has changed a few things in Phoebe release, namely the way successful invention jobs are handled in XML API, and Blueprints.yaml structure in Static Data Dump. As a result, current version of LMeve can neither load Phoebe Static Data, nor correctly show successful invention jobs.

I should be able to write a patch to support both changes until mid next week.

developers-portalWhy I haven’t done this sooner? Two reason really. Firstly, CCP has “commercially” started their Developer Portal recently – see it at https://developers.eveonline.com/. You can also set up shared secret required for EVE SSO there – feel free to use SSO in your own instance of LMeve! As a side effect, dev blogs regarding API and Static Data no longer appear on the Community page, and I simply missed the information about Static Data changes 🙁

Secondly, there was a lot happening IRL recently and I simply didn’t have the time to look at the Developers portal, start the game or develop LMeve.

Patch is coming, so stay tuned!

LMeve dev blog: tuning SQL performance and EVE SSO

Hello industry moguls and third party devs of EVE o7

First off, I’d like to say I respect Rixx Javix greatly for his last post about EVE industrialists and his respect towards them. Much appreciated, dear sir! But today’s dev blog post will be about three other things. First is a solution to the performance problems that Aideron Technologies instance of LMeve was having recently. Second is an announcement of a new feature, which thanks to @CCP_FoxFour will soon become widely available: ability to log in to LMeve using your EVE Online credentials (EVE SSO). Last but not least, I’ve added the new Blueprints.xml API endpoint.

Ohnoes, LMeve slows down!

I’ve got numerous complaints from my corp members recently about LMeve being slow and unresponsive. It was apparent on many subpages, so at first it was hard to pinpoint. Our database has grown over the past two years, and I thought this performance hit was caused by the amount of records we have accumulated:

lmeve-db

When you look at it, there is a few hundred thousands records there. But usually we only need to display the current month, and the rest has to be filtered out:

lmeve-graphs-1

This prompted me to have a look at how I filter out data by dates to show the current month only:

SELECT (...) FROM (...)
WHERE date_format(beginProductionTime, '%Y%m') = '${year}${month}'
AND (...);

At first glance it looks completely logical (and it actually works). I know the current month and year, so I only select records that have their year and month equal to the current one. But look closely what I’ve done there. I convert a DATETIME field “beginProductionTime” to a string which shows year and month in YYYYMM format, and then I compare this string with another string. As you probably know, comparing strings is much slower than comparing numbers (it’s why in databases you use indexes on text fields that need comparison). But I made it even worse; besides comparing strings, I converted DATETIME, and I did that for each and every record encountered by the query!

Here’s a piece of advice when you write an SQL query that sifts through a lot of data:

  • don’t run unnecessary functions or conversions on database fields that you use in WHERE clause
  • create indexes on TEXT and VARCHAR fields if you have to compare them

Each function which you put in there will have to execute for every record in the table, which will adversely affect performance.

When you take the two rules above into account, fixing my queries was pretty simple and straightforward. Instead of converting DATETIME values to string, I converted the compared string into a DATETIME:

SELECT (...) FROM (...)
WHERE beginProductionTime BETWEEN '${year}-${month}-01' AND LAST_DAY('${year}-${month}-01')
AND (...);

See? The values I compare beginProductionTime against are only evaluated once, instead of for every record, which means the whole SQL executes way faster than before.

Results:

Old query execution time: 37.4 seconds

New query execution time: 0.87 seconds

Op success! o7

Damn, I forgot my LMeve password again

lmeve-ssoWell, this kind of excuse will not be possible anymore, because thanks to @CCP_FoxFour and the limited EVE SSO trial, LMeve can now benefit from the SSO mechanism. CCP FoxFour explained SSO in detail in his dev blog here. How does this work?

  • LMeve redirects user to SSO at https://login.eveonline.com/
  • User logs in with his EVE Online login details and chooses one of the characters
  • EVE SSO redirects user back to LMeve
  • LMeve confirms with the SSO server that the login is valid
  • LMeve accesses the character name and ID and checks if that character is allowed to use LMeve (character must belong to the corporation configured in LMeve, and that character must also be linked to an LMeve account on “Characters” page. Users can do it on their own using their API keys.)

This is of course simplified exchange, because special token and secret values are also exchanged to make sure that nothing got spoofed on the way.

SSO feature for LMeve has already been developed and tested, but to set it up in your own instance of LMeve, you will need a special “client_id” and “secret” values. These values are similar to API keys, and you will be able to generate them  on the new 3rd party developer page (beta of this page working with Singularity server is already available).

New API endpoint

The last feature is the addition of Blueprints.xml API endpoint, which returns data about all the blueprints owned by a corporation, both originals and copies. Information retrieved from it contains ME and TE levels. Previously LMeve users had to input ME and TE manually for each blueprint owned by the corporation. With this new endpoint LMeve does all this automatically. This way all cost predictions made by LMeve always use up to date ME levels.

That’s all for today!

You can download the latest LMeve version from GitHub.

LMeve dev blog: security, guidelines and a handful of new features

In the last couple of weeks I focued on polishing LMeve. Especially that the project has a new contributor, who is both user and developer at the same time 😉 Say hello to @MarqAideron from Stay Frosty!

Security & CCP Guidelines

About two weeks ago @FuzzySteve suggested that EVE third party devs should use nonce values (also known as CSRF tokens) in all their web apps.

What’s that? Well, it’s a security mechanism, and a quite important one, too.

Let’s assume you are an admin of an open source web application (whether it’s a forum, blog or anything else). You log in to that application (of course using secure HTTPS protocol and a very secure password). Application then creates a random session identifier, verifies if user is connecting using secure connection and verifies the credentials. Then website then sends the session identifier back to the web browser in a cookie file. Each time you navigate around your app, web browser sends the cookie file to the server, so server knows which user he’s talking to.

Now let’s assume a hacker wants you to create an account for him in your app. Of course he can’t simply ask you to do it, but he knows your app, becuase it’s open source. He doesn’t know your password, or your session cookie, because it’s completely random. But he does know that adding a user in your app is done by this URL: https://your-app.com/?action=adduser&user=hacker_login&pass=1eK4Gaq4gj$g==

All the hacker needs to do, is make you open this URL while your web browser is logged to the application, for example by sending you an email with some embedded images. However, one of the images addresses is not an image at all: it will be the URL which creates a new account! What does your browser do when you open such email? It will attempt to load all images be default, including the crafted URL. It’s your browser, so it knows the session cookie for your application. Server will belive this is a genuine request coming from you, and application will create the account for the hacker! I don’t have to tell you what happens next. This is an attack known as Cross-site Request Forgery, or CSRF in short.

How to protect against this?

First: use POST in your forms instead of GET. This will make it harder for the hacker, because he won’t be able to use a simple URL to deceive the user (he will need a web form to send the crafted request).

Second: add a random hidden field in all your forms. When you draw the form for the user, you add a field with random “token” value. Save this token on the server side (it can be a session variable or a record in database). It should also have a short expiration timeout. so it cannot be used if it was somehow intercepted by the hacker. When user fills in the form and submits it, you verify if the value sent by the user is equal to the one you saved earlier. If both values are the same, request is considered genuine ,and otherwise it should be ignored.

CCP Bugartist followed on the conversation started by @FuzzySteve, and suggested a way to generate the CSRF tokens using openssl:

Don’t abuse the API, or else…

A few days later CCP FoxFour suggested adding an User-Agent field in third party apps, because instead of banning API (ab)users, CCP would prefer to contact the developer instead.

LMeve poller already used context setup in file_get_contents() so I simply added User-Agent field to it. Op success!

LMeve new features!

Ok, end of technicalities, let’s get down to business 😉 Last two weeks saw three new features added.

API character Self-register

This one was well overdue. I have mentioned it for the first time about a year ago. Why is it important? By default, LMeve doesn’t know which in-game characters belong to which person. This information is useful, because LMeve can then highlight the information regarding user’s characters, or simply limit the amount of information displayed only to these characters.

Previously only people with “Administrator” permission were able to link LMeve accounts to in-game characters, but with this new feature everyone can do it now. And it’s very simple, too.

First, input your account level API KEY first. If you don’t have a key ready, simply follow the link.

api-self-linkWhen ready, user clicks OK. LMeve will now access personal API (just this one time) to download the characters. Then it will compare the corporations these characters belong to with corporation that owns this particular instance of LMeve. When corporations match, LMeve will check if these toons are already visible in corporation API. If all checks are green, characters become linked to the account:

api-self-link2Simple, eh? And no administrator input is needed.

This will become even more important when EVE SSO becomes openly available, because once characters are linked to the account, users will be able to use their EVE Online password to log in to LMeve. there will be no need to remember another password!

Industry Facilities and Logistics

Everyone who’s done some industry in EVE knows just how important logistics are. Without ingredients corporation cannot invent or produce anything, so a constant flow of materials is the only way to keep productivity high (and this in turn directly impacts the amount of ISK earned).

Previously I’ve added a way for users to track the materials needed for their specific tasks, so users could refill their own labs easily. Most corporations however has a few dedicated pilots who do hauling and logistics, and they would benefit if they knew the materials needed by a specific assembly array, rather than specific character. With the information about Industry Facilities now available in the API, we only need to assign tasks to specific facility.

facilitiesWith this information available it’s then quite easy to calculate the materials required in each Lab:

facility-logisticsI’ve done the logistics and hauling recently, and you can believe me, it’s really useful!

What to build to become a real EVE Online industry mogul?

This is a tricky question. Profit on a single item is not enough to make a well informed decision, because some items, while profitable at single unit level, don’t sell very well. Other items have smaller margins, but because of their high sales volume, they give much higher profit. So what should you build? LMeve comes to the rescue.

profit-explorer

Under Database you can now find Profit Explorer and Profit Chart. The first one is very similar to normal Item database view, but it also calculates manufacturing cost and compares them with market prices. Please note that:

a) items must have their prices tracked (you must turn on “Fetch Prices” in Database for each item and each material; only minerals are turned on by default after LMeve installation)

b) items must have a bleuprint (so they can be built and their manufacturing cost can be calculated)

That’s all for today!

You can download the latest LMeve version from GitHub.

LMeve now compatible with Crius!

Last week was full of busy evenings and afternoons, but the result was worth the effort!

First off, LMeve is now compatible with Crius API endpoints:

  • /corp/IndustryJobs.xml.aspx endpoint has been changed to the new specs
  • /corp/IndustryJobs.xml.aspx endpoint has been added
  • YAML data from blueprints.yaml is now decoded and insterted into database

Secondly, new formulas are used to calculate the amount of materials required by industry jobs.

Lastly, YAML data is now used to populate “Traits” and “Bonuses” in Database. Oh, and I have almost forgotten, meta variants are now displayed as well:

lmeve-database-changes

As usual, code is available to download at GitHub.

Bug fixes

A friend of mine, Ubentobox, was trying to install LMeve on his host, and encountered a series of problems, caused by oversimplifications in my code. To prevent others from facing the same problems, I have fixed all the issues that he faced during deployment:

  • short PHP open tags ‘<?’ are replaced with long ones ‘<?php’
  • passwd.php is more intelligent now and is able to set or re-set admin password, without the need to manually edit the database
  • update_yaml.php is more intelligent now and can either insert all YAML data into database, or just the specified files
  • default cookiepath is now set to root ‘/’, so it’s easier to set-up LMeve in single site mode (cookiepath MUST be changed if multiple instances of LMeve are to be installed)
  • there is no longer need to rename camelCase table names in Static Data Dump to lowercase. LMeve now uses table names exactly as they apper in Fuzzysteve’s MySQL conversion.

The Future

Some new features I have planned will inculde:

  • Facilities XML endpoint, which will replace manually-edited Assembly Array feature
  • CREST Teams API
  • CREST public facilities API
  • Single Sign-On to connect characters to LMeve account (right now admins have to connect users to characters manually)