Steem Data Services (SDS) / Update Notice / Version 0.1.7b

in hive-102132 •  5 months ago  (edited)

Version Details

SDS will be updated to version 0.1.7b within the next few days. The new version is already in the process of being installed on sds0 (developer requests going to sds0 are automatically proxied to sds1 in the meantime).

Here are some of the most important new features and changes in this version:

  • New API Module transactions_api
     
    This API comes with a full transaction index and enables us to retrieve a block, transaction or block number by transaction_id. I worked really hard on this one to get the database size as small as possible. It's now only about 34G in size and still performs very well.
     
    For comparison, if one would create a regular database that contains an index on all transaction_ids of the Steem blockchain, the resulting size would be > 120G.
     

  • New API Module post_tags_api
     
    This one was already in preparation when the last version came out. Now a few of its API methods are finally ready to be used. It allows to quickly retrieve tag statistics like 'top used tags of all posts/comments in the blockchain' and 'top active tags of all active posts/comments'.
     
    It also returns a field count for each tag, so it can be used to build some nice daily/weekly statistics too. More methods will be added in future.
     

  • New API Method witnesses_api/getWitnessVotesSummary
     
    This method returns a list of all incoming witness votes for the given witness (including the total proxied vesting shares per voter). The resulting rows are sorted by influence descending.
     
    It was backported to the 0.1.7a already to have way faster backwards proxy and witness votes searching capabilities on SteemWorld.

  • New Field last_reply in all Feed Lists
     
    The field is now available in all feed list results (for root posts only) and it contains the timestamp of the last created reply for each post. If there are no replies, it will be returned as '0'.
     
    Thanks to @moecki for the great ideas ;)

Update

I needed to start a full 're-parse' operation, therefore I could use the opportunity to extend the new post_tags_api already in this version:

Installation Status

  • [x] Installing the new version on sds0
  • [x] Live testing the version on sds0
  • [x] Redistributing to sds and sds1 servers

Have a good one! Steem on ~ 🚀


If you like what I do and you want me to be your Steem witness,
please vote for @steemchiller on steemit.com/~witnesses or steemworld.org/witnesses.


Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Cool! Danke für deine Arbeit! 👍

New API Module post_tags_api

Die API ist schon sehr interessant. Das erspart mir das Durchlaufen von Posts. Bin gespannt, welche Methoden du da noch einbaust. TagsByCommunity und TagsByAuthor wären sehr nützlich für eine Community- und Autorenseite.

I worked really hard on this one to get the database size as small as possible.

Würde mich schon interessieren, wie du das gemacht hast. Könnte aber auch verstehen, wenn du hier nicht alle deine Algorithmen offenbaren möchtest.
Bedeutet das, dass du für jede API eine eigene Datenbank führst, die mit jedem Block aktualisiert wird?

TagsByCommunity und TagsByAuthor wären sehr nützlich für eine Community- und Autorenseite.

Da sich die Version noch in der Testphase befindet und ich aufgrund eines festgestellten Bugs heute sowieso einen kompletten 're-parse' starten muss, bevor sie in Produktion gehen kann, werde ich bei der Gelegenheit die post_tags_api noch etwas erweitern. Mehr Details dazu demnächst ;)

Bedeutet das, dass du für jede API eine eigene Datenbank führst, die mit jedem Block aktualisiert wird?

Ja, der Parser durchläuft die Blöcke und ruft entsprechende Events in den über die Config aktivierten Modulen auf (soweit das jeweile Modul nicht bereits bei einer höheren Blocknummer ist). Die Module können natürlich auch untereinander verknüpft werden, um Daten anderer APIs direkt abrufen zu können.

Jedes Parser-Modul verwendet normalerweise mindestens eine eigene Datenbank, die aber auch untereinander verknüpft (attached) werden können. Über die Config kann man auch festlegen, dass eine Datenbank für mehrere oder sogar alle Module verwendet werden soll, aber das würde ich aus Performance- und Wartungs-Gründen nicht empfehlen.

Es lassen sich auch Daten aus anderen Modulen direkt in SQL-Abfragen joinen, daher hat es deutlich mehr Vorteile die Datenbanken zu trennen und so unfragmentiert wie möglich zu speichern.

Die Aufteilung bringt zudem auch hinsichtlich der Verminderung der Redundanz sowie der im Notfall erforderlichen Wiederherstellung Vorteile. Allerdings wirst du dir dazu im Vorfeld wohl sehr viele Gedanken über die Art und Weise der Aufteilung und anschließender Verknüpfung (durch die SQL-Abfragen) gemacht haben. Dabei den Überblick zu behalten, ist nicht ohne.

Für den privaten Bereich habe ich ein paar mehr oder weniger komplexe Datenbanken erstellt und mit entsprechenden Abfragen und Codezeilen versehen. Insofern habe ich ungefähr eine Vorstellung davon, welchen Aufwand du mit der effizienten Speicherung der Daten der gesamten Blockchain hattest/hast.

Mehr Details dazu demnächst ;)

Bin gespannt. :-)
Heute hatte ich noch einen neuen Gedanken. Oftmals interessiert mich, bei welchen Posts neue Kommentare eingestellt wurden und würde danach gern sortieren, damit man auch neue Kommentare in älteren Posts nicht übersieht. Für jeden einzelnen Post bekäme man das mit den Replies auch raus. Allerdings ist dies für eine Post-Liste (z. B. aus der feeds_api) nicht ganz so "unaufwendig", da man erst zu jedem Post in der Liste die Replies abrufen müsste.
Wäre es anhand der Daten in deinen Datenbanken (unkompliziert) machbar, in der Response (z.B. nach feeds_api.getCommunityPostsByCreated) auch den Timestamp des letzten Kommentars zurückzugeben?

Wäre es anhand der Daten in deinen Datenbanken (unkompliziert) machbar, in der Response (z.B. nach feeds_api.getCommunityPostsByCreated) auch den Timestamp des letzten Kommentars zurückzugeben?

Mir gefällt deine Idee und ich denke, dass ich ein Feld last_reply in allen Feed-Listen für Root-Posts problemlos zurückgeben könnte. Wahrscheinlich wird das aber erstmal nur für neue Kommentare (Erstellungszeitpunkt) funktionieren, da ich sonst einen weiteren Index für das Feld updated (das den Zeitpunkt der letzten Änderung enthält) brauchen würde.

Ich schau mal, was ich da machen kann.

Das wäre doch sehr gut. Ich denke, es wäre nicht weiter tragisch, wenn es zunächst nur die neuen Kommentare funktioniert. Das wird sich dann ja nach und nach ändern.
Wenn du es eingebaut hast, wäre eine Info gut, damit ich weiß, wie das feld last_reply zurückgegeben wird, wenn es keine (oder keine neuen) Kommentare gibt.

Thanks for the latest update. You are very kind. Hopefully, Steemit will be more successful. The presence of steemworld.org really helps me in Steemit. We are waiting for the next update, @steemchiller
😊😊😊

Thanks for the compliment

You always release information that is very important for us to know the Steem Data Service. thank you very much for providing more information regarding Steem Data Services (SDS) / Update Notice / Version 0.1.7b .

I always appreciate your work. Thank you for this.

You are doing an excellent job, you will see that all hard work pays off, congratulations on the upcoming steemit sds updates.

yes Mr @steemchiller always release information that is very important for us to know the Steem Data Service. thank you very much for providing more information regarding Steem Data Services (SDS) / Update Notice / Version 0.1.7b .

These are great updates sir and i believe with this new upgrades and feautures, SDS would be a better working interface.

Thank you for this again, you are always doing a great Job which is very noticeably and well appreciated and i see no reason why i shouldnt vote for you.

I voted for you as a witness some weeks ago😊.

awesome! great work!

Servus Chiller,

gibt es einen Grund, warum bei getPostsByTagsText und getActivePostsByTagsText der Parameter searchText nicht optional ist?

Hintergrund ist, ich möchte die Posts von zwei (vielleicht auch mehr) Tags auslesen, also z.B. alle aktuellen Posts bei denen die Tags #deutsch und #steemexclusive vorhanden sind.

Moin, ich habe geplant mehrere Tags für alle relevanten Methoden der feeds_api zuzulassen, aber bin leider noch nicht zur Umsetzung gekommen.

An sich könnte ich den Parameter aber wirklich ohne viel Aufwand optional machen. Schaue ich mir heute Abend mal an. Wahrscheinlich könnte ich das sogar noch problemlos mit in die 0.1.7c aufnehmen.

ohne viel Aufwand hört sich ja gut an. Bin schon gespannt, was die 0.1.7c mitbringt. Möcht ja nicht wissen, wieviel Stunden du mit SDS verbracht hast.

Mercy für die Antwort!

Ich habe es jetzt erstmal so gelöst, dass der Suchtext "" für alle Methoden mit Tags-Parameter funktioniert und auf 'Suchen ohne bestimmten Text' umschaltet. Richtig optional kann ich es an der Stelle nicht so einfach machen, da noch Parameter nach dem Suchtext kommen könnten.

Das Ganze sollte bereits auf dem Testserver (sds0.steemworld.org) und App-Server (sds1.steemworld.org) laufen ;)

Das klingt doch schon mal sehr gut. Danke! Da juckt es mir doch in den Fingern, meine Workarounds gleich mal umzubauen... :-)

Schaut gut aus, sehr gut sogar, DICKES Danke!

Was ich bist jetzt gesehen habe, die API gibt Tags ab der sechsten Position nicht mehr her, SDS schon - sehr schön.

Was mir noch aufgefallen ist, nachträglich reineditierte Tags mag SDS anscheinend nicht, kann das sein?

Sehr schön! Geänderte Tags sollten normalerweise berücksichtigt werden. Falls dies nicht der Fall sein sollte, muss ich da nochmal ran die Tage.

Hello. Please help me restore access to my sister Eugene's account https://steemit.com/@gmenka, here on steam - @gmenka . Her account has been hacked and power down is enabled.

Hi, please go to https://steemitwallet.com/recover_account_step_1 and follow the instructions on that site. @gmenka will then receive an email with further instructions for recovering the account. Important is that Eugene uses the same email address that she used when signing up on Steemit.

Thank you. It's already been done. Unfortunately, the letter has not arrived yet. Time is running out and I'm afraid that the attacker will withdraw money from the wallet

The email should arrive immediately as long as the used email address matches the one from the registration on Steemit. If it does not arrive, it may be the wrong email address used when requesting the account recovery.

And if it so happened that she doesn’t remember on which mail she registered the account or, perhaps, she was blocked, she just started writing on Steam a very long time ago. is it possible to unlock the sister's account using passport data or can she make a video call and find her identity?

I informed the Steemit team about the case. So, she will hopefully receive the email soon ;)

Hello Steemchiller.

There is a new user who has received the confirmation

Hi,
Good news! You have successfully created the account.
Username: rebbrownie233
E-mail: rebecca1441007@gmail.com
Phone number: ‪+63 918 797 3878‬
To get your login details and Keys for Steemit.com, go and sign in to the Steemit wallet at
https://steemitwallet.com
Cheers,
Your friends at Steemit

But receives this error message when attempting to login

image.png

Who does one contact about this?

Hi, I think it's not a good idea to share the user's contact details here in a comment... Anyway, it looks like the account was not successfully created (maybe it's still in progress), because there exists no account with name 'rebbrownie233' in the blockchain at this point.

Let's wait a bit and see if it will be created later today. In case it does not get created within the next few hours, I will contact the Steemit team ;)

  ·  2 months ago (edited)

Hi @steemchiller,

I was curious if you know of a way to get the historic vesting fund in order to convert historic curation rewards in vests to dollars. I've realized that using your api call I can get historic curation rewards much faster than with the function I made a few months ago. Thanks for making it!

Sincerely,
@cmp2020

Update:

I've been trying to solve this problem all day, and figured out how to get the conversion rate using the virtual op "fill_vesting_withdraw" at your api call for a range of virtual ops (calculating the block to start at based on the date). But this method is extremely slow (took about 50 seconds just to get the rate). I was curious about how you and steemit convert curation rewards in vests to steem power.

Hi @cmp2020, there is still no good solution for retrieving the historic values available. As I was asked this question a few times already, I'm planning to provide a new method that can offer a better/faster way in the near future.

I was curious about how you and steemit convert curation rewards in vests to steem power.

The conversion from VESTS to SP just uses the current values from the blockchain. As we receive rewards in VESTS, whose worth increases over time, we acually earned more in terms of SP than what was shown in the past. Hard to say, if it would be more correct to always use daily conversion rates.

Hello @steemchiller
need your help.

Have you made any API which returns list of Recent posts inside a community ?
For Example i want to get Latest post over Steemit Pakistan How can i get that ?

Hi @hassanabid, you could probably use the Feeds API for that:
https://sds0.steemworld.org/feeds_api

For example, this method would return the last 100 active posts for a community:
https://sds0.steemworld.org/feeds_api/getActiveCommunityPostsByCreated/hive-172186

Here you find the description for that method:
https://sds0.steemworld.org/describeMethod/feeds_api.getActiveCommunityPostsByCreated

But there are many more methods available. In case you have any questions later on, just let me know ;)

Thanks man you are hero ❤️😘

I just want to get link of that post only anyway to do that? Tried split up json but its to difficult How can we apply limit to get links only.?

The limit parameter can be used to limit the number of rows, but there is currently now way to define the returned columns via API. You would need to create a small script that locally converts the data into your desired format.

But I like the idea of having the possiblity to convert directly via API. I will think about adding some kind of converter to support different formats in the future.

Yeahhh columns limiter could be a great thing :)
Do let me know whenever you release that.
Thankyou so much ❤️

This wonder jo .Thanks for the information. very useful for us to know the development and it is great achivevement .@steemchiller

wow appreciated ! Thank you for your work! 👍

The API is very interesting. This saves me from looping through posts. I'm curious what other methods you use. Tags By Community and Tags By Author would be very useful for a community and author site

Please stop copying other users' comments!

Thanks for the information. very useful for us to know the development @steemchiller

Thank you for your update

This API comes with a full transaction index and enables us to retrieve a block, transaction or block number by transaction_id. I worked really hard on this one to get the database size as small as possible. It's now only about 34G in size and still performs very well.

It might be surprise for us great .

this is an excellent job,
curious for update

Your work is most appreciated! Keep it up

Hi
Do you perhaps know what translation I should use with the date and time value.
Date and time fields are represented by an interger. I am developing in c#
Thank you

Hi, I'm always working with Unix timestamps. I never used C#, but I think this site should be helpful for you:
https://www.delftstack.com/howto/csharp/how-to-get-the-unix-timestamp-in-csharp

Thank you very much. Exactly what i was looking for

Hello Steemchiller. Your steemworld, is by far the most helpful application for steem. I have a question about password security. You are one of the most trusted people on Steem, but anyone can be hacked. What happens if Steemworld gets hacked? If I have put in my active key, will it be in danger? I have been told never to enter your active key into any site. How does steemworld protect our keys that we enter? I am not very knowledgeable about the technical aspects so forgive me if this is silly question. Thanks

Hi there!

If I have put in my active key, will it be in danger?

SteemWorld does not store any keys on any server. In case of using the private posting or memo key on the site, they are locally stored in your browser cache (encrypted). The strict CSP of the SteemWorld domain prevents the site from sending out data to unprivileged servers. So, when using a modern browser, there is nothing that can happen in this regard.

Of course, SteemWorld can not prevent yourself from getting hacked. If you store your private keys unencrypted on your pc (in a text file or so), there is always the risk of someone getting access to your system and just copying the keys.

I always recommend users to install the Steem Keychain browser extension and use only that for signing transactions that require the active key. Keychain stores your keys encrypted with a master password on your machine and is therefore the most secure solution.

When using Steem Keychain on SteemWorld, the signing takes place locally within the extension and even SteemWorld does not have access to any of your higher keys.

Hello sir I am Admin of Steem India community
Which is an Indian community and I want your support for the growth of the community it would be great if you visit us . Thank you and I have a great day ahead!

Hi
How do I get/calculate the reputation of an account?

Don't wary. Got it sorted

Ok, in case you still need help, here is how the official steem-js library solved it:
https://github.com/steemit/steem-js/blob/master/src/formatter.js#L157-L167

Sorry, keep on bugging you:(
I cannot find out how to calculate or find the SteemPower of an account.
This is what I am busy with:
https://steemtools.azurewebsites.net/

You need to calculate the amount of STEEM per vesting share with help of the global props (get_dynamic_global_properties) and then multiply it by the number of VESTS.

steem_per_share = total_vesting_fund_steem / total_vesting_shares

Thanks. Been struggling to find the get_dynamic_global_properties address.
Then, the Limit and Offset, cant seem to get it right.
Lets say I want to return 10 rows starting at position 10
https://sds0.steemworld.org/accounts_api/listAccounts/created/DESC/?/?/10/10
No matter what I use I get the same data back everytime.

Been struggling to find the get_dynamic_global_properties address.

If you want to solve it with SDS, you could just use the
https://sds0.steemworld.org/steem_requests_api/getSteemProps,
which returns a field 'steem_per_share'.

No matter what I use I get the same data back everytime.

The 'listAccounts' method is not described in detail yet, because it might change in future. I probably will add a replacement without the filter param for the next version.

Currently it would work with this one:
https://sds0.steemworld.org/accounts_api/listAccounts/created/DESC/{}/false/10/10

Perfect. Thank You very much

mmm No I think it may be calculating something else. If I look at the wallet, one have Steem, Steem Power and Steem Dollars.
I want the Steem Power. In my case it is 271.539

Thank you

Thanks you @steemchiller for the information.

Hi, I would like to say a thousand thanks to you sir who always support me in any condition. I don't know how to repay what you have given me.

I can only wish you good health and continued success in everything you do.

I always try to say thank you on Discord but I haven't had a response from you in a long time.

THANKS YOU
Regards

Hi friend, you were very helpful when my account was blacklisted before.

The same situation has now happened to another friend of ours in our community. We would be very happy if you could help with this.

Thank you very much in advance

https://steemit.com/hive-187187/@habercitr/please-help-me

Hi, the account was successfully recovered and just needs to be removed from the blacklist, right?

Yes friend my , the account has been recovered. It should be removed from the blacklist 🙏

Ok, I will let the team know ;)

Thank you very much 🙏🙏🙏😊

My friend, thank you very much for your attention and help. Our friend's account has been activated again🙏😊

Upvoted! Thank you for supporting witness @jswit.
Please check my new project, STEEM.NFT. Thank you!
default.jpg

Thank you @steemchiller for your information.
I want to work here as a active member.
Please approved me.

Guten abend! And nice meeting you!
It looks like you are just the right person to help me.
I've got two Steemians who wants to return to the platform.
There is no problem with the wife's account but the husband cannot login with his keys.
@graviton
Please can you help?

Hi @patjewell, I don't see anything wrong with the account @graviton. The owner key history shows only one entry, so his original keys should still work here. He could try to sign in on steemworld.org, just to check, if it is related to some kind of bug on steemit.com

  ·  last month (edited)

Thank you! I appreciate you coming back to me. I will ask him to check again

hello friend.help me.i have lost my account key .i have lost,but i failed. my account name is @asamkana1

Hi @jal08, if you don't have access to @asamkana1's Master Password or Private Owner Key, there is no way to recover the account. Nobody other than yourself knows your keys, therefore there is nothing we could do in such case.

Always keep a copy of your Master Password or Private Keys in cold storage (offline location)!

my phone has been restarted, so all the contents of my documents have been deleted, the account password can't be recovered, this happened three months ago.. so I created another steemit account.. please support from you. so I can join steemit, like tan other friends..

Thank you for this very useful information my good friend, you always do a great job, a very good job and can last a long time in this steemit and I am happy if you help us for full support to steemit players, thank you very much for providing information more about Steem Data Services (SDS) / Update Notifications / Version 0.1.7b.
And I want to continue to learn to maintain good steemit like you my friend.

#steemworld