02/07/2019 incident Report

in likwid •  3 years ago  (edited)

The bug took place the 2nd of July 2019 at around 18:00PM.

We apologize to all affected customers. We are making sure it will not happen again.

Service should be back to normal within 4h after this post publish time.

@samotonakatoshi was the first person to give notice about the incident.


First of all, likwid was not hacked.

The reason was a bad coding practice. To be specific, a bad error handling.

All started when https://dev.steemit.com rpc node gave a random 502 bad request error. Likwid was set to re-invoke itself in such cases. That ended up with multiple Likwid instances running in parallel, which created an exponential endless loop of self-invoking likwid instances. Having that many asynchronous instances made impossible for the database and fail-safe features to prevent multiple redundant payouts.

Final result and lost balance

We found out about the issue quick enough and the server was brought down within the first 20min after the issue-trigger.

The following table contains the different likwid customer accounts who got multiple transfers.

@taifkhansent back54.33919.452
@threespeakwallet sent back15.7255.628
@dongkong sent back1.4944.176
@kabasakal sent back27.6189.886
@ura-soul paid back70.45725.222
@mahdier sent back153.47254.938
@wehmoen sent back394.068141.059

For all affected accounts, we would appreciate if you can send back the lost funds. Please let us know when funds are sent back. Thanks a lot in advance.


For the sake of transparency and compliance, please find below the foresincs snippet code that has been used (nodeJS + dsteem library) to identify the lost founds:

async function forensics () {
    var transfers = []
    let history = await client.database.call('get_account_history', ['likwid', -1, 1000])
    var duplicates = []
    transfers = history.filter((x) => {return  (x[1].op[0] == 'transfer' && x[1].op[1].from == 'likwid')}).map((x) => x[1].op[1])
    const unique = [...new Set(transfers.map(item => item.to))]

    for (let i = 0; i < transfers.length; i++) {
        let el = transfers[i]
        let _duplicates = transfers.filter((x) => { return x.to == el.to && x.amount == el.amount})
        if (_duplicates.length > 1) {
            duplicates.push(...new Set(_duplicates))
    let final_duplicates = []
    final_duplicates.push(...new Set(duplicates))
    for (let i = 0; i < unique.length; i++) {
        let author = unique[i]
        let _transfers = final_duplicates.filter((x) => x.to == author)
        if (_transfers.length > 1) {
            sbd_transfers = _transfers.filter((x) => getCurrency(x.amount) == 'SBD')
            if (sbd_transfers.length > 1) {
                total_sbd_debt = sbd_transfers.reduce((a, b) => { return { amount: parseFloat(a.amount) + parseFloat(b.amount) } })
                console.log(parseFloat(total_sbd_debt.amount) + ' SBD')
            steem_transfers = _transfers.filter((x) => getCurrency(x.amount) == 'STEEM')
            if (steem_transfers.length > 1) {
                total_steem_debt = steem_transfers.reduce((a, b) => { return { amount: parseFloat(a.amount) + parseFloat(b.amount) } })
                console.log(parseFloat(total_steem_debt.amount) + ' STEEM')
Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

From what I can see I was paid twice for one post but then not paid for a later post.

This one was paid twice: https://steemit.com/@ura-soul/pkoshmbj
(25.222 SBD + 70.457 STEEM)

This one has not been paid: https://steempeak.com/@ura-soul/nkuppeoy
(23.575 SBD + 68.140 SP)

So I make that an outstanding balance that I need to pay of: 1.647SBD + 2.317 STEEM

It would be easier for me to just pay the final amount in SBD as I Powered up the liquid STEEM already. Let me know what you want to do.

Hello, could you please join our discord likwid channel? https://discord.gg/YzAb4dS

Sure, I am there.

I just sent you direct message in discord.

Payout has been released 3e2da86ee60621ce69a63aa371b402f1a89e61de

@likwid will you please check my payout i have not get my payout from last 3 days ?

Been trying to contact you in discord. Would you please join? Thanks a lot in advance.

All pending payouts have been now released.

I have returned the amount !


Tnq 🙂

This post earned a total payout of 0.325$ and 0.244$ worth of author reward which was liquified using @likwid. To learn more.

  ·  3 years ago (edited)

I think you got another bug, since you keep paying people who got flagged/downvoted, and your payout does not seem to take that into account.

I have typed to discord channed. Still waiting for may response.

Still waiting for my payment

Rewards have been released.

bot is mistaken

Hey there, I noticed you might have missed a post payout from my threespeak about 10 hours, could you check it for me? Thanks ^^

this one

screenshot of my wallet, seems to be missed.