Thread Tools Display Modes
06/25/20, 10:25 AM   #1
Keldor
 
Keldor's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2015
Posts: 101
ESO-Database Game Data API for AddOn Authors

Hello everyone,

tomorrow I will release the new version of my ESO-Database Client and it includes a first version of the Game Data Export AddOn. This AddOn currently stores the positions of Chests, Thieves Troves and Psijik portals.

I imagined that this data could be useful for many AddOn authors, so I made the data available via API. The advantage of this is that several hundred players per day will use the AddOn starting tomorrow with the release of the new client to contribute data. There are already some data available from the Client and AddOn beta testers

You can find the API documentation and endpoints here:
https://game-data.eso-database.com

For the future I have plans to integrate books and harvest nodes as well. This could be interesting for Harvest Map and Lore Books

What do you think about this idea to make the data available for all AddOn authors?
If you have some wishes or ideas for other data, feel free to ask me :
  Reply With Quote
06/26/20, 02:43 AM   #2
Micke2nd
AddOn Author - Click to view addons
Join Date: Mar 2017
Posts: 43
Originally Posted by Keldor View Post
This AddOn currently stores the positions of Chests, Thieves Troves and Psijik portals.
I imagined that this data could be useful for many AddOn authors
Hi Keldor,
the only use case I know is to visualize their locations, what HarvestMap does.
If I would need the locations outside the game, I would reuse HarvestMaps files.

What another use case do you have in mind ?
  Reply With Quote
06/26/20, 03:01 AM   #3
Baertram
Super Moderator
 
Baertram's Avatar
WoWInterface Super Mod
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 4,912
Originally Posted by Micke2nd View Post
Hi Keldor,
the only use case I know is to visualize their locations, what HarvestMap does.
If I would need the locations outside the game, I would reuse HarvestMaps files.

What another use case do you have in mind ?
I guess this is the main usecase.
"Collect ifnormation by multiple users without having to install/use different addons, but just use 1 data collector storing the data online where addond evs can grab them for their addons".

This is what I understood.
Thanks for the effort Keldor.
It's a good idea and I hope it will work.

One problem would be to tell the users to use your addon constantly. It will create SavedVariable files that grow a lot as you collect data which is already in multiple other addons you have named yourself -> means BIGsv files -> Long loading screens.
Means 2x big SV files (e.g. harvest map and your colelctor addon= 2x longer loading screens).
They will recognize this and disable your collection addon then.

The other problem I see would be addons not using the same method or data base you are collecting and need to "transfer" your data again to "their internal" data. Which might them stick at their own build in data collectors, included into the addons which the users use to "visualize" the map pins etc.
So in the end it might be a nice idea but not work for some, or just being ignored as their own methods are working and they do not need or want to invest time where not needed.

But having the data shared online to be able to read them from there, even for websites, is a nice feature!

Zone data and anmes always redundantly in the tables!
But I wouldn't ALWAYS include the zoneData like names into all the collected information!
It blows up the files and data tables unnecessarily.
Just create a zonedata table with the names maybe and just reference in your other data on the zoneIds then.
Will make fetching data for the thieves, psijic, etc. "slim" as you can always use the build in ESO API functions to get the zoneName ingame.
Or use your other zoneData table as lookup.
All you need is the zoneId e.g.

And I know many addons already who only would need the zoneId + zoneNames in multiple languages. So collecting this data with your collector "as standalone API/table" would be even nice and could be used. e.g. for LibZones, which many other addons re-use then.


Chests
Do you somehow assure that the collected chests data by multiple users, in the same zone, does not report like "20 chests near by each other", where it actually is only 1 chest spawning at slitghtly different locations?
Or do the addons just grab your total data and need to assure this?
Again a matter of data size here then. You will have 20+ entries but only 1 is needed.

P.S.
Thanks Keldor for working on this!

Last edited by Baertram : 06/26/20 at 03:16 AM.
  Reply With Quote
06/26/20, 04:55 AM   #4
Keldor
 
Keldor's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2015
Posts: 101
Originally Posted by Micke2nd View Post
Hi Keldor,
the only use case I know is to visualize their locations, what HarvestMap does.
If I would need the locations outside the game, I would reuse HarvestMaps files.

What another use case do you have in mind ?
The advantage of the whole thing is that the data automatically ends up in the database without having to ask the individual users to contribute the data. It is not obligatory to use my data but for one or the other developer it could be a good place to go.


Originally Posted by Baertram View Post
One problem would be to tell the users to use your addon constantly. It will create SavedVariable files that grow a lot as you collect data which is already in multiple other addons you have named yourself -> means BIGsv files -> Long loading screens.
Means 2x big SV files (e.g. harvest map and your colelctor addon= 2x longer loading screens).
They will recognize this and disable your collection addon then.
Since the data is uploaded after every /reload or logout, I have limited the amount of data to be stored in the AddOn, so the SV files shouldn't get really big


Originally Posted by Baertram View Post
Zone data and anmes always redundantly in the tables!
But I wouldn't ALWAYS include the zoneData like names into all the collected information!
It blows up the files and data tables unnecessarily.
Just create a zonedata table with the names maybe and just reference in your other data on the zoneIds then.
Will make fetching data for the thieves, psijic, etc. "slim" as you can always use the build in ESO API functions to get the zoneName ingame.
Or use your other zoneData table as lookup.
All you need is the zoneId e.g.

And I know many addons already who only would need the zoneId + zoneNames in multiple languages. So collecting this data with your collector "as standalone API/table" would be even nice and could be used. e.g. for LibZones, which many other addons re-use then.
I think the thing with the zones is a good point. I will revise that again and only add the zone id. For this there is another API call for the zones, which I also have in the database.


Originally Posted by Baertram View Post
Chests
Do you somehow assure that the collected chests data by multiple users, in the same zone, does not report like "20 chests near by each other", where it actually is only 1 chest spawning at slitghtly different locations?
Or do the addons just grab your total data and need to assure this?
Again a matter of data size here then. You will have 20+ entries but only 1 is needed.
I will publish the AddOn here on ESOUI soon, then you can have a look at the methods how to export the data. I'm also happy to take some advice here to improve or extend the whole thing

Thanks in any case for the questions and suggestions. As I said, the basic idea behind this project is to support developers with their great AddOns which are already available on ESOUI.

Keldor
  Reply With Quote
06/26/20, 05:17 AM   #5
Kyoma
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 125
For LoreBooks it'll depend on how the data gets collected as it isn't always as simple as collecting a harvest node. Taking the uesplog addon for example would lead to many erronous and duplicate data.
  Reply With Quote
06/26/20, 05:40 AM   #6
Keldor
 
Keldor's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2015
Posts: 101
Originally Posted by Kyoma View Post
For LoreBooks it'll depend on how the data gets collected as it isn't always as simple as collecting a harvest node. Taking the uesplog addon for example would lead to many erronous and duplicate data.
If you like I can have a look at your lore books exporter functions to include this part to the addon. The AddOn is also available on Github. If you like you can provide the lore books export functionality

https://github.com/KeldorDE/ESO-Data...a-Export-AddOn
  Reply With Quote
06/26/20, 05:54 AM   #7
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,566
Nice idea! Don't have any immediate use-cases for it, but will keep it in mind for the future.

I have one suggestion though. You should provide the mapId (GetCurrentMapId) together with the coordinates. That way an addon can determine where the coordinates are originally from.
  Reply With Quote
06/26/20, 05:58 AM   #8
Keldor
 
Keldor's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2015
Posts: 101
Originally Posted by sirinsidiator View Post
Nice idea! Don't have any immediate use-cases for it, but will keep it in mind for the future.

I have one suggestion though. You should provide the mapId (GetCurrentMapId) together with the coordinates. That way an addon can determine where the coordinates are originally from.
Thanks for the suggestion. I will add this with the next release
  Reply With Quote
06/26/20, 10:13 AM   #9
Micke2nd
AddOn Author - Click to view addons
Join Date: Mar 2017
Posts: 43
ok, its possible to implement, so have fun

To support also the WHY question:
Originally Posted by Baertram View Post
I guess this is the main usecase.
"Collect ifnormation by multiple users without having to install/use different addons, but just use 1 data collector storing the data online where addon devs can grab them for their addons".
That already exists.
HarvestMap "collects by multiple users without having to install different addons" Chests, Thieves Troves, Psijik portals and Harvest nodes.
Why not to ask Shinni if he would add a API ? If - and that's still open - if another addon would need that data.

Originally Posted by Baertram View Post
But having the data shared online to be able to read them from there, even for websites, is a nice feature!
Yes, that's a advantage HM doesn't support. Ok and how many webpages need that data ? For what? I havent found one. But let's say suddenly one will be created. For 1 user that effort ?

Originally Posted by Baertram View Post
One problem would be to tell the users to use your addon ...
Yes, because except the support fact, users have no advantages.

Originally Posted by Keldor View Post
The advantage of the whole thing is that the data automatically ends up in the database without having to ask the individual users to contribute the data.
More automatically than to click the bat file from harvestmap ? Very unlikely.
If Lore Books doesnt have this upload/sync actually, I would have started with its data.

Last edited by Micke2nd : 06/26/20 at 10:19 AM.
  Reply With Quote
12/29/20, 04:42 PM   #10
DewiMorgan
 
DewiMorgan's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 14
Could it gather and share usage data that HM does NOT provide?

For example: I'm interested in making an addon that modifies maps to show where cliffs are. Putting their locations in the map is easy enough manually, but heights are another issue. Crowdsourcing the falling damage taken (before armor etc) would give an idea of how high the cliffs are in a way that's maybe more useful to map readers than knowing how many feet high a cliff is. As well as just making for a very amusing heatmap of where people most often yeet off cliffs.

Another example: I've written an addon which needs a translated whitelist of searchable items. I was considering crowdsourcing so that any item people hit E to use, which wasn't already in my addon, would save its location and name in various languages. We could very rapidly build an item-translation database that way. Perhaps even more importantly, we could find cases where items are inconsistently translated (where eg two items, one metal and one wood, both called "barrel" in English, get called "barrel" and "drum" in some other language). This would not easily be found by devs otherwise.

I'm sure there are plenty of other crowdsourceable data types that would be super useful to devs; while the answers here seem negative, might that just be that for your first API release, you happen to have selected the very few data types that turn out to already be well-tracked and crowdsourced already?
  Reply With Quote
12/30/20, 03:16 AM   #11
Keldor
 
Keldor's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2015
Posts: 101
Originally Posted by DewiMorgan View Post
Could it gather and share usage data that HM does NOT provide?

For example: I'm interested in making an addon that modifies maps to show where cliffs are. Putting their locations in the map is easy enough manually, but heights are another issue. Crowdsourcing the falling damage taken (before armor etc) would give an idea of how high the cliffs are in a way that's maybe more useful to map readers than knowing how many feet high a cliff is. As well as just making for a very amusing heatmap of where people most often yeet off cliffs.

Another example: I've written an addon which needs a translated whitelist of searchable items. I was considering crowdsourcing so that any item people hit E to use, which wasn't already in my addon, would save its location and name in various languages. We could very rapidly build an item-translation database that way. Perhaps even more importantly, we could find cases where items are inconsistently translated (where eg two items, one metal and one wood, both called "barrel" in English, get called "barrel" and "drum" in some other language). This would not easily be found by devs otherwise.

I'm sure there are plenty of other crowdsourceable data types that would be super useful to devs; while the answers here seem negative, might that just be that for your first API release, you happen to have selected the very few data types that turn out to already be well-tracked and crowdsourced already?
Hi,

thanks for your ideas.

I still have the TODOs for collecting resources, books, etc. on my list. Unfortunately, I haven't had time for that yet.
The cliff thing is more a very specific part of data collection, right? I think this is to much overhead for "fun data"

I've also been considering building an Item DB, since I already have a lot to do with Item IDs in the ESODB anyway, through the recipes, furnishings, and now more recently the Set Collection. But here I have the problem with the time and effort, because I want to do other things before.
  Reply With Quote
12/30/20, 03:25 PM   #12
DewiMorgan
 
DewiMorgan's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 14
Originally Posted by Keldor View Post
The cliff thing is more a very specific part of data collection, right? I think this is to much overhead for "fun data"
Oh, fersure: my point wasn't "you should implement these specific things for me!" but just to show that even I, a total noob, had already felt the need for crowdsourced data. There will be good things and bad things to gather, and if the possibility's there, I'm sure you'll get a zillion suggestions for both!

So my main point was just to say that perhaps people are only being negative, because they aren't seeing the full power and scope of the things an API like this could offer in the future?
  Reply With Quote
12/31/20, 04:40 AM   #13
Keldor
 
Keldor's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2015
Posts: 101
Originally Posted by DewiMorgan View Post
Oh, fersure: my point wasn't "you should implement these specific things for me!" but just to show that even I, a total noob, had already felt the need for crowdsourced data. There will be good things and bad things to gather, and if the possibility's there, I'm sure you'll get a zillion suggestions for both!

So my main point was just to say that perhaps people are only being negative, because they aren't seeing the full power and scope of the things an API like this could offer in the future?
Oh, then I actually misunderstood I had understood it as a feature request.
  Reply With Quote

ESOUI » Developer Discussions » Dev Tools » ESO-Database Game Data API for AddOn Authors

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off