LMeve dev-blog: Bugfixes and POS tracking improvements

Hi space friends!

More and more of you have already tried LMeve, which I am incredibly grateful for. This means more people are testing the app and I get more feedback, which is awesome. I also get a lot of questions which I try to answer the best I can and in a timely fashion.

Remember: by trying and using LMeve you help developing it!

Few players also caught some bugs, that I’m happy to announce, have been properly squashed in the 0.1.52 release.

  • Bug fix in db.php “Notice: LM_dbhost is not defined” – added $LM_dbhost to global declaration in db.php
  • Bug fix in Database – “function eregi() is deprecated” – replaced with preg_match()
  • Bug fix in YAML updater – changed ‘DROP TABLE’ to ‘DROP TABLE IF EXISTS’ to prevent errors if the table was not created yet.

One bug was found by my trusty Aideron Technologies beta testers:

  • Non-recurring Tasks (singleton) bug fix – one-time tasks did not display correctly due to a mistake in SQL query

What’s new besides Bugfixes?

  • Poller now fetches StarbaseDetails.xml and CREST industry cost indexes

The first API feed is used by POS tracking code to display accurate and up-to-date fuel levels in LMeve Inventory. The second CREST feed provides NPC cost index to the build cost estimation formula (I currently use an average system index, so you might pay more or less than what LMeve uses for price estimates).

In the next release I will add a “Industry home system” field in Settings, where you will choose your home system. LMeve will then use this system’s cost index in all it’s calculations.

  • New POS management screen – displays fuel amounts and how long it will last
  • There is now a notification when any tower has less than 48 hours of fuel left

new-pos-screen

  • Tech III invention added. You can now add Tech III invention tasks and get price estimates for Tech III
  • New global setting for the default Relic type for use with Tech III Invention (Intact, Malfunctioning or Wrecked)

Download LMeve here

https://github.com/roxlukas/lmeve/

If you are updating from a previous release, please remember to create new tables included in this release using /data/schema-delta-0.1.51-0.1.52.sql

Try-before-you-buy

Well, LMeve is free software, so you don’t have to buy it 🙂 (donations are of course welcome!). But let’s get back to business. Installation requires some technical knowledge, so to make it easier for you guys to try it out, I have prepared a pre-installed Virtual Machine with LMeve, which runs on free VMware Player. You can download the image here: https://pozniak.pl/lmeve-vm/ 

  • it contains LMeve 0.1.52
  • static data is updated to Scylla
  • after booting it shows the IP address to point your web browser to:

lmeve_vm

Torchwood Archives is now hosting caldariprimeponyclub.com

Hi spacefriends!

A lot of you probably know T’Amber’s ship painting project, that used to be available at caldariprimeponyclub.com.

jeremy

Well, our friend T’Amber had some hard time making it work with models and textures hosted on the CCP CDN (for some reason CORS headers were set for some people, but were missing for others, so Jeremy was not able to load texture files).

But there is no such thing as impossible for two dedicated capsuleers 😀

Together we’ve been able to restore Jeremy’s core functionalities by moving it to my host, and configuring it to use LMeve’s built in proxy for model and texture files originally hosted by CCP CDN.

To access Jeremy, simply head to http://pozniak.pl/caldariprimeponyclub.com/

jeremy-quafe-dominix

LMeve dev blog: POCO details page and client list!

In November 2013 CCP has allowed high sec NPC Customs Offices owned by Interbus to be destroyed and then replaced with player owned ones. Previously only low-sec Customs Offices were vulnerable. Of course having POCOs makes a very nice passive source of ISK (you don’t have to do anything, apart from an occassional call to arms to defend them). But have you ever wondered how much you earn through your POCO network or who is using them?

In a previous update of LMeve I have added the ability to visualize which POCOs earn the most ISK, and in this iteration I continue the trend to show even more details. You can now choose a specific POCO and see a Details page for it. This page shows how much this specific Customs Office earned in the last 30 days – counting between 30 days ago and today, income in the last calendar month, and in the current one, so you can see trends).

What’s another new thing is the list of your customers. Corp Wallet Journal entries for planetary taxes show the client’s name and planets ID, so by running an aggregate query on that you can see:

  • who is using a specific POCO
  • how often they use it
  • how much they pay in taxes (so you can approach them and offer a discount in exchange for let’s say blue status)

Note: due to this information being obtained from corp Wallet Journal, it won’t show characters who have 0% tax rate, as these people will only pay the NPC tax, and no taxes to your corp at all (so no records are added to corporation Wallet Journal).

PI new planet detail screen

If you have ideas for more features in LMeve, please feel free to contact me. The best way is to raise an Issue on project’s Github page: https://github.com/roxlukas/lmeve. Choose an appropriate category (bug fix, enhacement or question) and write down what you need, or what needs fixing. I can’t guarantee that all requests will result in that feature being added to LMeve, but if the feature looks useful, won’t take forever to code and most importantly, the information required to make it is in the EVE API, it has a high chance of being included in one of the future releases.

Tweetfleet Slack

Just one more thing. Ashterothi, ex-member of Aideron Robotics and well-known host of podcasts such as High Drag and Hydrostatic Podcast, has recently created a Slack team for Tweetfleet. Slack is generally a communications platform aimed at companies, but it has a very nice real time chat (plus a dedicated app for iOS & Android). It is very useful, and a dedicated #devfleet channel also exists, so feel free to drop by and chat with fellow players, CCP devs and third party devs. It’s really fun!

To get invited, simply head to https://www.aideronrobotics.com/tweetfleet/ and fill in the form.

Tweetfleet Slack itself is here: https://tweetfleet.slack.com/

LMeve dev blog: Big POCO update coming

Just as promised in the previous LMeve dev blog, I have recently added the ability to track income from individual planets. Moreover, with the new percentage bars, you will know what are your best planets simply by looking at the table. To check how much a specific planet earned you in the last 30 days, simply hover your mouse above that planet’s name to see a tooltip with the exact number.

lmeve-poco-update

But this is not the only thing added in this release. Some pages require crunching a lot of data, which means they render rather slowly. Most of this data doesn’t change very often, so it seems to be a good idea to generate them once, and then save a cached copy. Then every time users come back to this page, a cached version is displayed, saving everyones time.

One of the pages, the Profit Chart can have several hundred records, which need their manufacturing and invention prices calculated, and it can load for as long as few minutes (depending on individual configuration of course). I used a different approach here: a lazy loader. Instead of one big 2 minute request, I display the page immediately, and as user is scrolling, more data is being loaded into the table in packs of 10 rows.

How to link POCOs with planets & income?

It’s not trivial, but far from impossible. Let’s have a look at columns returned by the /corp/CustomsOffices.xml.aspx API:

<row itemID="101296791423" solarSystemID="30004295" solarSystemName="Keba" reinforceHour="19" allowAlliance="True" allowStandings="True" standingLevel="-10" taxRateAlliance="0" taxRateCorp="0" taxRateStandingHigh="0" taxRateStandingGood="0.05" taxRateStandingNeutral="0.07" taxRateStandingBad="0.1" taxRateStandingHorrible="0.15" />

There is POCO itemID, which stays the same in all API endpoints, there is solarSystemID and system name. And of course the tax settings. But there is nothing about planet itemID here. Maybe /corp/Locations.xml.aspx endpoint will reveal a bit more?

<row itemID="101296791423" itemName="Customs Office (Keba I)" x="58598019193.2515" y="-5937030740.62145" z="-3890977687.95432" />

Okay, so now we have x, y, z coordinates, and POCOs name. In the first version of POCO management screen I simply cut planet name out of this using a regular expression. But to link it with other data we still need the planet itemID. How do we find it? Well, POCO is orbiting a planet, right? And we have cooridnates of all planets in the Static Data Export database – in the mapDenormalize table. What are we waiting for?

CREATE FUNCTION `findNearest`(`x1` DOUBLE, `y1` DOUBLE, `z1` DOUBLE, `solarSystemID1` INT) RETURNS int(11)
 READS SQL DATA
 RETURN (
 SELECT a.itemID FROM
 (SELECT SQRT(POW(x1-x,2)+POW(y1-y,2)+POW(z1-z,2)) AS dist,itemID
 FROM mapDenormalize
 WHERE `solarSystemID`=solarSystemID1
 ORDER BY dist ASC
 LIMIT 1) a
 )

40271955

Bingo! The function returns the closest solar body, in this case, itemID of the planet. We just have to JOIN two other tables from SDE: invNames to convert the planet ID we just got into planet name, Keba I, and invItems to find it’s typeID (Temperate, Barren, Oceanic, etc.).

Ok, now on to income for this specific POCO. For this, we will need  /corp/WalletJournal.xml.aspx endpoint, which contains all the transactions of a corp. The records we are looking for will have refTypeID equal to 96 and 97 (import/export taxes) and the planet itemID will be in argID1 field. Once all joins are in place, we simply have to filter last thirty days and SUM all the amount values.

As usual, the current LMeve version can be downloaded from https://github.com/roxlukas/lmeve

New third party page: evelexicon.com

@Fuzzysteve (Steve Ronuken, member of CSM #9 and the guy behind the most popular SDE-to-MySQL and SQLite conversions) has surprised the community today with a very nice simple website:

What does it do? You’ve guessed it, it’s a simple wiki-like vocabulary for EVE Online. Everyone is welcome to add new definitions to it. How do you log on? Simple: Eve Lexicon is using EVE SSO, so go on, log in using your EVE Online credentials, and add some definitions!

The site is written in PHP with a tiny bit of JQuery to make it pretty. EVE SSO integrations are now possible thanks to the new Third Party Developers Portal. A sample SSO library for developers (also by @Fuzzysteve) is available at github.com.

LMeve dev blog: Catching up with Phoebe. Bonus: setting up SSO

Good news! Last monday I had a day off, so I finally caught up with all outstanding tasks, including making LMeve Phoebe-compatible. In the meantime, I started tinkering with the following features:

Vagrantfile, bootstrap.sh

These two files make it easier to set up a host for any app. Trent Bartlem suggested to me on Github to include these files back in August. While the files themselves are pretty straightforward to prepare, I have to install LMeve from scratch somewhere to make sure I included all the necessary packages required to run LMeve. It’s on my to do list, and it should happen sooner than soon™.

Page Caching

Another idea I have is to make the longest loading pages cached (and maybe – just maybe – refreshed in the background, for example by the poller). Code to make this kind of caching possible is already written, and two sample pages (Inventory, Profit Chart) have been added as well. I use AJAX call to request the cached page, and if  cached version is not available (or has expired) I display a “Loading…” sign and prepare a refreshed version of the page. I was thinking about adding a “Force refresh” button, so users can force LMeve to refresh a cached page.

I still evaluate how this improves performance, but it looks pretty good and should become public soon.

PoCo module

lmeve-pocoLast one on the To Do list is expanding PoCo module. It has been recently revamped with features such as displaying last month’s income and current month income prediction. Judging by the talk I had with Dracoth Simertet from RvB (they have a huge PoCo network) and CCP FoxFour, there is a demand for a tool for managing PoCo networks. But what features would you need to monitor your PoCos?

  • Last month income per PoCo (currently only a sum for entire corp is displayed)
  • Planet type the PoCo is orbiting
  • Number of interactions
  • Client list per PoCo (by corp, by char)
  • Reinforced status
  • Do you have more ideas? Please leave a comment if you do!
    • We are only limited with what’s available in the API

SSO integration

I have mentioned a few times already that LMeve can benefit from EVE SSO, but some players were not sure how to set it up. Don’t worry! I’ve got you covered! Here’s a quick guide.

  • Secondly, click “Create new application”
    developers-02
    developers-03
  • Fill in the form. Application Name can be for example “LMeve – <your corporation name here>”
  • Now make sure to fill in Callback URL field correctly. If your LMeve address is https://myhostname.com/lmeve/, then the correct callback will be https://myhostname.com/lmeve/ssologin.php
  • Save the changes.

Once the application has been created, open it again by clicking the “View application” button, and write down these two values:

  • Client ID
  • Secret Key

developers-04

they will be required to complete setup on the LMeve side.

  • Log in to your web server host, open up config/config.php in your LMeve directory.
  • Find these variables, enter your callback URL and the values written down in previous step:
//use EVE SSO - see https://wiki.eveonline.com/en/wiki/EVE_SSO_Documentation
$SSOENABLED=TRUE; 
$SSO_REDIRECT_URL='https://myhostname.com/ssologin.php';
$SSO_CLIENT_ID='sso_client_id';
$SSO_CLIENT_SECRET='sso_client_secret';
//Auth server can be either login.eveonline.com for Tranquility, or sisilogin.testeveonline.com when trying to use Sisi.
$SSO_AUTH_SERVER='login.testeveonline.com';
  • Make sure characters are connected to LMeve usernames. This is crucial for the SSO to work!
    • Go to “Characters” module in LMeve and check if your characters are there.

lmeve-sso

Now you are ready to use EVE SSO to login to your LMeve instance!

LMeve dev blog: Phoebe support delayed

I have a very quick announcement today. CCP has changed a few things in Phoebe release, namely the way successful invention jobs are handled in XML API, and Blueprints.yaml structure in Static Data Dump. As a result, current version of LMeve can neither load Phoebe Static Data, nor correctly show successful invention jobs.

I should be able to write a patch to support both changes until mid next week.

developers-portalWhy I haven’t done this sooner? Two reason really. Firstly, CCP has “commercially” started their Developer Portal recently – see it at https://developers.eveonline.com/. You can also set up shared secret required for EVE SSO there – feel free to use SSO in your own instance of LMeve! As a side effect, dev blogs regarding API and Static Data no longer appear on the Community page, and I simply missed the information about Static Data changes 🙁

Secondly, there was a lot happening IRL recently and I simply didn’t have the time to look at the Developers portal, start the game or develop LMeve.

Patch is coming, so stay tuned!

Evernus – third party market tracking app

I have recently stumbled upon a third party program, which in my opinion, deserves a spotlight.

Evernus is a market tracking app made for Windows, but it is open source and written in Qt, which means with a little effort it can be compiled for Linux and Mac as well (there’s actually a .dmg ready for download). This is something worth noting, because there is almost no EVE apps for Mac or Linux at the moment.

What does it do?

Evernus is basically a market/margin trading tracker application, so it is focused on prices comparison and buy/sell order tracking. It takes character skills into account when calculating profits and taxes, so it is pretty accurate. Of course to track orders and automatically update your character’s skills, you need to supply the app with your API key.

If your character doesn’t have all trade skills at level 5 yet, but you would like to experiment and see if training them to 5 is worth it, you can manually tune all relevant skill levels as well.

evernus-character-full1Of course Evernus shows pretty graphs, so at a glance you know if you’re making profit or not.

evernus-basic-statistics-full1
Don’t like graphs? Want exact values? You’re covered, simply view it as a table.

evernus-assets-full2 evernus-orders-full2Evernus can use price data from several sources. First, it’s your EVE client’s market exports. Secondly, it can scrape client cache (but please mind CCP considers it a breach of EULA, and Evernus shows a proper warning, if you try to enable this feature), or from internet. Lastly, you can also import your data from other apps such as EVE Mentat.

What’s also a very useful feature is in-game browser integration. Evernus can run a tiny web server on your PC, and you can access it directly from the inside of the EVE client, simply by opening in-game browser and pointing it to your own computer’s address, plus a port number which you set in options.

evernus-igb-full1

One more thing

Evernus is being actively developed by Pete Butcher, a member of polish EVE Online community. New features come out literally DAILY. Moreover, Pete contacted me, and LMeve integration is coming soon as well (Evernus will be able to access LMeve’s features like Industry Task tracking and item manufacturing costs).

All in all, if you’re a station trader, Evernus is a tool really worth looking into.

LMeve dev blog: security, guidelines and a handful of new features

In the last couple of weeks I focued on polishing LMeve. Especially that the project has a new contributor, who is both user and developer at the same time 😉 Say hello to @MarqAideron from Stay Frosty!

Security & CCP Guidelines

About two weeks ago @FuzzySteve suggested that EVE third party devs should use nonce values (also known as CSRF tokens) in all their web apps.

What’s that? Well, it’s a security mechanism, and a quite important one, too.

Let’s assume you are an admin of an open source web application (whether it’s a forum, blog or anything else). You log in to that application (of course using secure HTTPS protocol and a very secure password). Application then creates a random session identifier, verifies if user is connecting using secure connection and verifies the credentials. Then website then sends the session identifier back to the web browser in a cookie file. Each time you navigate around your app, web browser sends the cookie file to the server, so server knows which user he’s talking to.

Now let’s assume a hacker wants you to create an account for him in your app. Of course he can’t simply ask you to do it, but he knows your app, becuase it’s open source. He doesn’t know your password, or your session cookie, because it’s completely random. But he does know that adding a user in your app is done by this URL: https://your-app.com/?action=adduser&user=hacker_login&pass=1eK4Gaq4gj$g==

All the hacker needs to do, is make you open this URL while your web browser is logged to the application, for example by sending you an email with some embedded images. However, one of the images addresses is not an image at all: it will be the URL which creates a new account! What does your browser do when you open such email? It will attempt to load all images be default, including the crafted URL. It’s your browser, so it knows the session cookie for your application. Server will belive this is a genuine request coming from you, and application will create the account for the hacker! I don’t have to tell you what happens next. This is an attack known as Cross-site Request Forgery, or CSRF in short.

How to protect against this?

First: use POST in your forms instead of GET. This will make it harder for the hacker, because he won’t be able to use a simple URL to deceive the user (he will need a web form to send the crafted request).

Second: add a random hidden field in all your forms. When you draw the form for the user, you add a field with random “token” value. Save this token on the server side (it can be a session variable or a record in database). It should also have a short expiration timeout. so it cannot be used if it was somehow intercepted by the hacker. When user fills in the form and submits it, you verify if the value sent by the user is equal to the one you saved earlier. If both values are the same, request is considered genuine ,and otherwise it should be ignored.

CCP Bugartist followed on the conversation started by @FuzzySteve, and suggested a way to generate the CSRF tokens using openssl:

Don’t abuse the API, or else…

A few days later CCP FoxFour suggested adding an User-Agent field in third party apps, because instead of banning API (ab)users, CCP would prefer to contact the developer instead.

LMeve poller already used context setup in file_get_contents() so I simply added User-Agent field to it. Op success!

LMeve new features!

Ok, end of technicalities, let’s get down to business 😉 Last two weeks saw three new features added.

API character Self-register

This one was well overdue. I have mentioned it for the first time about a year ago. Why is it important? By default, LMeve doesn’t know which in-game characters belong to which person. This information is useful, because LMeve can then highlight the information regarding user’s characters, or simply limit the amount of information displayed only to these characters.

Previously only people with “Administrator” permission were able to link LMeve accounts to in-game characters, but with this new feature everyone can do it now. And it’s very simple, too.

First, input your account level API KEY first. If you don’t have a key ready, simply follow the link.

api-self-linkWhen ready, user clicks OK. LMeve will now access personal API (just this one time) to download the characters. Then it will compare the corporations these characters belong to with corporation that owns this particular instance of LMeve. When corporations match, LMeve will check if these toons are already visible in corporation API. If all checks are green, characters become linked to the account:

api-self-link2Simple, eh? And no administrator input is needed.

This will become even more important when EVE SSO becomes openly available, because once characters are linked to the account, users will be able to use their EVE Online password to log in to LMeve. there will be no need to remember another password!

Industry Facilities and Logistics

Everyone who’s done some industry in EVE knows just how important logistics are. Without ingredients corporation cannot invent or produce anything, so a constant flow of materials is the only way to keep productivity high (and this in turn directly impacts the amount of ISK earned).

Previously I’ve added a way for users to track the materials needed for their specific tasks, so users could refill their own labs easily. Most corporations however has a few dedicated pilots who do hauling and logistics, and they would benefit if they knew the materials needed by a specific assembly array, rather than specific character. With the information about Industry Facilities now available in the API, we only need to assign tasks to specific facility.

facilitiesWith this information available it’s then quite easy to calculate the materials required in each Lab:

facility-logisticsI’ve done the logistics and hauling recently, and you can believe me, it’s really useful!

What to build to become a real EVE Online industry mogul?

This is a tricky question. Profit on a single item is not enough to make a well informed decision, because some items, while profitable at single unit level, don’t sell very well. Other items have smaller margins, but because of their high sales volume, they give much higher profit. So what should you build? LMeve comes to the rescue.

profit-explorer

Under Database you can now find Profit Explorer and Profit Chart. The first one is very similar to normal Item database view, but it also calculates manufacturing cost and compares them with market prices. Please note that:

a) items must have their prices tracked (you must turn on “Fetch Prices” in Database for each item and each material; only minerals are turned on by default after LMeve installation)

b) items must have a bleuprint (so they can be built and their manufacturing cost can be calculated)

That’s all for today!

You can download the latest LMeve version from GitHub.

LMeve now compatible with Crius!

Last week was full of busy evenings and afternoons, but the result was worth the effort!

First off, LMeve is now compatible with Crius API endpoints:

  • /corp/IndustryJobs.xml.aspx endpoint has been changed to the new specs
  • /corp/IndustryJobs.xml.aspx endpoint has been added
  • YAML data from blueprints.yaml is now decoded and insterted into database

Secondly, new formulas are used to calculate the amount of materials required by industry jobs.

Lastly, YAML data is now used to populate “Traits” and “Bonuses” in Database. Oh, and I have almost forgotten, meta variants are now displayed as well:

lmeve-database-changes

As usual, code is available to download at GitHub.

Bug fixes

A friend of mine, Ubentobox, was trying to install LMeve on his host, and encountered a series of problems, caused by oversimplifications in my code. To prevent others from facing the same problems, I have fixed all the issues that he faced during deployment:

  • short PHP open tags ‘<?’ are replaced with long ones ‘<?php’
  • passwd.php is more intelligent now and is able to set or re-set admin password, without the need to manually edit the database
  • update_yaml.php is more intelligent now and can either insert all YAML data into database, or just the specified files
  • default cookiepath is now set to root ‘/’, so it’s easier to set-up LMeve in single site mode (cookiepath MUST be changed if multiple instances of LMeve are to be installed)
  • there is no longer need to rename camelCase table names in Static Data Dump to lowercase. LMeve now uses table names exactly as they apper in Fuzzysteve’s MySQL conversion.

The Future

Some new features I have planned will inculde:

  • Facilities XML endpoint, which will replace manually-edited Assembly Array feature
  • CREST Teams API
  • CREST public facilities API
  • Single Sign-On to connect characters to LMeve account (right now admins have to connect users to characters manually)