Delivery failures increase over the June 11/12 weekend

[Arwen Serpente]

Hi All :)

I've been watching my delivery status on my items and over the June 11/12 weekend there was a definite increase in "Delivery Partially Failed". My magic boxes are working and have delivered the same items successfully to other customers. In the cases of the failures, the orders were only for my items (not a mixed cart with other Merchant items which often results in that "Delivery Partially Failed" message). Some of the items in the orders were delivered, others failed. In some cases, an order of just one item shows up as "Delivery Partially Failed" which is rather strange since I'm not sure how 1 item can "partially fail".

I have left messages for the customers to check but have not yet heard back. I have also filed support tickets for the failed deliveries.

Posting here in the forum to see if other Merchants are seeing an increase (or if you've checked) and to give a head's up to the Commerce Team (aside from the support tickets) that something seems to be off (moreso than our usual issues with deliveries).

[Darrius Gothly]

Just cuz I'm tenacious ... and still hunting for that "common thread" ... by chance did you look at the profiles of the buyers that got partially failed deliveries to see how "old" they are? Was it mostly newbies, oldbies, in-betweenies?

[Arwen Serpente]

Yup, I've been contacting them all morning. Not new avatars, some date back to 2007. Only spoken to one so far and she did not receive the product and it is unclear if she will reorder so ---> lost revenue.

On a side note, when these deliveries fail, they are still counted as "Units Sold" in the Top Selling Reports. So that report is highly inaccurate given the rate of failed deliveries.

[Paladin Pinion]

Just as a data statistic, I've never had a failed delivery on any product I sell, ever. From what I've read here, I'm probably lucky.

More data: I only sell a few things, they sell at a casual but steady rate,and  my Magic Box is on a private estate that almost never needs to be restarted. I suspect it's my lower volume of sales that makes me lucky. I don't think my Magic Box ever acquires a queue.

I'm looking forward to the new inventory transfer mechanism regardless, I think that will solve a lot of issues for everyone.

[Darrius Gothly]

I have duplicates of my three boxes. One pair holds most of my items, one pair holds 50+ items that are currently unlisted because I'm waiting for Multiple Storefronts (they are products that just don't "fit" my other stuff), and one pair that just does the SEO items. Each set is on a private Estate, each Estate is low-lag and seldom needs restarting. Both sets are also placed at 4000+ m altitude. (I've found that lag gets much lower at those nosebleed heights.)

I seldom have failures .. meaning maybe one per month, one every two months. However they do occur.

@Arwen - Daggone it, I was hoping to find out it was newbies that got the most failures, but since it's primarily middle- and oldbies, that theory is shot down in flames too.

Would be educational to find out what the Lag is like where the recipient is located when a delivery is attempted. No matter how much we optimize the delivery side, if the receiver side is standing in a Sim with TD of 0.1 .. death, doom AND destruction are sure to follow.

[Josh Susanto]

My failure rate remains utterly stable this week from last week. That is: I get at least one failed delivery notice every day, but I don't much care. Considering that I have over 600 full perms products and they're priced between 0L and 9L, you can probably imagine the volume of orders I'm getting.

I must, again, reiterate that I'm absolutely convinced that any pattern in my delivery failures is totally unrelated to the number of items in my only box, what the items are, or how large the order is.

Delivery failures occur repeatedly to some avatars, and never to others.

The problem is absolutely account-specific by buyer.

[Arwen Serpente]

I just looked at my various stats for delivery failures vs. delivery partially failed:

~ delivery failures: 5 since the Marketplace opened in Sept 2010 out of thousands of sales, so I'd say that's pretty decent ratio.

~ delivery partially failed: 63 since Sept 2010, scattered over every month (more this past weekend than usual which is why I posted and sent support tickets). In these cases, most of these orders must have had something from another Merchant fail since my items show as delivered.  Normally, I hear from a customer immediately if they don't receive something so that I can help with a redelivery (assuming it is available for redelivery). I do find that I am "redelivering" pretty frequently (several times per week) but it may be due to a customer declining the object, crashing when it is being sent, or some other glitch unrelated to magic boxes.

I have spoken to 3 of the customers who had "delivery partially failed" from my store this weekend: in every case, the delivery failed and they were refunded the lindens. One had bought 6 things and actually was unaware of the issue until they checked order history; the second went to my store to purchase inworld since the delivery failed; the third was sending birthday gifts so the "moment" was lost and most likely will not reorder. Waiting to hear from the rest that I contacted.

[Darrius Gothly]

@Josh & @Arwen:

The more I've chased this down, the more stats I've collected and reports I've read ... I'm really coming to the conclusion that the things we do to optimize our magic boxes may be helpful in a small way, but it's akin to brushing the teeth on a dead horse.

This conclusion *may* be why LL chose to rush forward Direct Delivery ahead of any other gripe/bitch/complaint echoing from the Merchant Body. If they arrived at the conclusion that nothing the Merchants could do would resolve the problems on the Recipient end of the transfer, and that a high majority of delivery failures are due to Recipient issues ... then implementing a means to stuff the items into their Inventory whether they're online or not would make a lot of sense.

However, no matter WHAT the answer is, no matter the reason or cause ... it would have been nice for someone to SAY SO! *shakes head*

C'mon Brooke, we are not idiots out here. For heaven's sakes, we managed to survive untold upheaval in the migration, adapt to radically different structures, tools, layout and methods ... and we manage to build, texture and script stuff that's complex and sophisticated. Do we HAVE to beg for tidbits of information like babies crying for a bottle and a diaper change? Please?!?

Just a simple "We've identified the primary cause of delivery failures to be intermittent and accidental disconnections or issues with Recipients accepting the delivered items. Therefore we are progressing on a design that will ensure proper delivery direct to their inventory without failure." (Then, after we shot ToySoldier with a tranq dart .. things would be okay.)

 

 

Sorry Toy! *grin*

[Josh Susanto]

>it would have been nice for someone to SAY SO! *shakes head*

Thanks for making that clear.

It seems to me that Brooke has actually been more responsive since people (um, me, at least) have made a point of emphasizing to Brooke that the core of the mass of complaints is not what has gone wrong or what has been done about it, but a simple pattern of failure to acknowledge problems.

But, appreciating Brooke's position in regard to the other Lindens, I think it's imperative that we continue to emphasize this point.

[Darrius Gothly]

There's two ways to get people to follow you. You either establish yourself as a strong, reliable, trustworthy person with strong leadership skills and impeccable integrity ... or you carefully and patiently describe the goals, explain the rationale behind choosing those goals, and "sell it" to the people. (There is a third way, but it involves masses of mindless dolts and sheep that will follow anything that moves. However I feel pretty safe declaring the population of Second Life as anything BUT mindless sheep.)

Considering that Brooke's predecessors pretty much destroyed any hope of trust being developed (although she has done a much better job than many expected), and it seems that her actions are dictated by voices and directions above her a lot of the time, I think it's highly unlikely a loyal following will develop under Scenario #1 above.

Therefore that leaves Scenario #2 .. explain everything.

It's not a sure-fire fix, but it's my opinion that it has a much higher chance of success than saying nothing at all.

[Josh Susanto]

Or just don't get people to follow you.

That could work, too... right?

[Yazmine Tsarchon]

Hi there, I'm relatively new to the marketplace, but until recently, I had very few failures. Since yesterday, I had nothing BUT delivery failures. I have sent off my items free of charge just to keep the customer happy. I've now unlisted everything because I don't know what else to do. I have two magic boxes, unfortunatley, I don't have a totally private sim. One is in a Linden home and the other is on my friends rental land (she rents a 1/4 sim).. it's in the sky box, but only about 300 meters up.

I even tested with an alternate account of mine. They were not online even, but the delivery failed to them as well.

Has it gotten better for any of you? How many magic boxes do you have down? Should I put more down? Higher up? This is very frustrating...

[Arwen Serpente]

Hi Yazmine, the "delivery partially failed" continues for my store. Had 2 failures last night:

1) Customer tried to buy 3 items in a shopping cart (all from my store and all in different magic boxes, so they would not be cycling through, just delivering one item from each one) - it failed. This morning, she tried again, this time as individual purchases (1 item per shopping cart transaction) and all 3 were successful.

2) Customer tried to buy 1 item from my store in a cart with 3 other items from other Merchants - it failed. They tried again moments later - this time with 1 item from my store and 1 from another store - successful.

My magic boxes are in a skybox (not sure of the altitude). I have a total of 375 items on the Market divided into 11 magic boxes. The sim I am on is practically lagless (especially since I am in a skybox and not near any neighbors).

Not sure if that's helpful to you. I hope so

[Yazmine Tsarchon]

Hi Arwen, I don't mean to hijack your thread... but I have a question again after reading your reply. Well, I only have 37 items at the moment... lol.. nothing close to the volume you have

The magic boxes I have are all exactly the same... the same 37 items in them. You think I should be splitting them up or doing as I have been?

I'm going to keep testing ideas, I put a 3rd one down now even higher up! LOL... tested again with my alt account and failure again.

[Arwen Serpente]

Don't worry, it's not hijacking - lots of Merchants are experiencing a variety of delivery issues and sharing experiences helps.

As far as number of items in a magic box --> 37 items in one box should be fine. For testing purposes, you could always divide them up, but it really shouldn't be necessary.

One thing, Yazmine, I just looked for your store in the Marketplace and it doesn't show up when I search your name (perhaps you use a different account/name for your store).

[Yazmine Tsarchon]

I unlisted everything for a while since I was testing, now I listed it again, I will give it a resting period and try again... hahaha, it's frustrating and really disappointing too since I'm just getting started. I was hoping this would be a way to get some feedback and so on before opening an inworld store. Now I feel rushed to get something in world going right away before I'm ready to make the leap. :matte-motes-frown:

[Arwen Serpente]

I just checked and your store is up and running on the Marketplace now.  Crossing my fingers that your deliveries start to work!

[Josh Susanto]

I had a brief drop in the number of orders, but everything continues to deliver just fine here, with my 620 F###ing items all jammed in the same box.

Scientifically speaking, I almost wonder whether the rest of you shouldn't just try things my way this week.

[Arwen Serpente]

lol! maybe so!

As I write this, the same customer who ordered a cart of 5 items (3 mine) and had the transaction fail twice over the weekend is trying to send the items again. I was inworld and watching the box...this particular one has been working all day long until this transaction (meaning this same box has delivered all day). Both the sender and recipient are inworld/online. There is no lag in the sim - I'm at 70+ fps.  Simultanteously, other orders are being placed and going thru from different boxes (I actually can see the status change from "queued" to "being delivered" to "delivered". But not this one. The status is currently "being delivered" but it is hanging there for the past 30 minutes or so. I have a feeling it is going to fail again. I'll report back when something happens one way or another.

Edited to update:

As of 5:33pm slt, the order is still "being delivered" to the customer. As a test, I logged on with my store alt and gifted one of the items to myself. It is also hanging in "being delivered" state - the magic box did not turn green and no item has been received. This box IS delivering all other items in it properly. I am wondering if it is the "gift" part of the equation that is somehow failing. I say this because I logged onto "Arwen", went to another sim, rezzed a new magic box, placed just this line of items in it and sent yet another one to myself. While I stood there, the box never turned "green" as it should (I can only assume the box on the other sim did), because immediately I received the item. I cannot for the life of me figure this out.

Update 7:24pm: Both purchases failed. Mine to my store alt (lol, I got a refund on Arwen) and the one that was from a purchaser to another person as a gift.

[Arwen Serpente]

Darrius, I think the common thread is "gifting".

After reviewing my failed and partially failed deliveries, the common thread is not the item sent, or the number of items, it is that the item(s) are being sent as gifts. I now have instances of "Gift giver" sending items to "Giftee" --> failed delivery.  When that same "Giftee" comes back and repurchases the same items for themselves --> success.  I just did it myself: I gifted an item to my store alt --> failed delivery; I sent the same item to myself --> success. 

This occurance has increased in the month of June. Items with failed/partially failed delivery status are trending at double a normal month and we are only half way through June.

[Arwen Serpente]

It's now June 17 and the delivery failures continue. Now it seems they are focused on one line of products and failing when sent as gifts or when sent to self. The following steps have been taken:

1) Reboxed all items
2) Bought a new Magic Box
3) Deleted existing Magic Boxes that contained this line of items
4) Placed the newly boxed items in the new Magic Box (only these items; there are 12, but only 9 have ever been listed - I did not change that)
5) Unlisted, then relisted on Marketplace (did not create new listings, just went to manage inventory and checked them, unlisted, then rechecked them and listed. They are NOT being listed as "new")
6) Resynced Marketplace
7) Tested sending to my store alt ArwensCreations Resident: Still failed

My other boxes and clothing lines continue to sell and to deliver to customers. I don't know what else to do, but it seems that there something preventing the Marketplace from communicating with whatever Magic Box I use, containing this particular line of items.

I welcome any advice the group can provide!

There is also a JIRA for this kind of issue:

https://jira.secondlife.com/browse/WEB-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Please add your comments and experiences if this is affecting you!

[Darrius Gothly]

---

Josh Susanto wrote:

Or just don't get people to follow you.

That could work, too... right?

---

Umm .. LOL Yeah, that works too. Although being in a leadership position while attempting to shake loose followers is not exactly what could be considered "wise business practices".

[Darrius Gothly]

---

Arwen Serpente wrote:

Darrius, I think the common thread is "gifting".

---

That would be a very bad thing if true. Based on what you've written though, it sure is sounding suspect. I also had another idea of what might cause delivery failures ... but for non-gifting scenarios.

When a customer purchases a number of items from several different merchants, they are queued so items from a single merchant get sent out at a specific rate. In prior tests we determined the max rate that items from a single merchant go out is never faster than once every 13 seconds.

But what we didn't consider is what happens when items from two different merchants are sent out simultaneously. Worse yet, what about items from three or four different merchants sent at the same time. Because the magic boxes are all on different Sims (most likely) there's a very good chance all the magic boxes would get the "send" command all at once .. and perform a mini-DDoS attack on the recipient's client. In effect, the recipient would receive two or more Delivery IMs all at once, and might very well drop one or two of them.

In any event, the upcoming DD (if successfully implemented without massive mortality rates) will fix these problems. It's just a shame that we have to toss out a system that used to work just fine but stopped working with the new Marketplace .. just to totally rewrite a new system with untold and as yet unforeseen consequences and bugs.

[Arwen Serpente]

I honestly can't figure out what's going on. I was having the delivery failures to giftors (originally), then it morphed into giftors and self-buyers of just a certain line of products, and then last night, I had two sales of the items that kept failing - one as a gift and one as purchase to self - and both succeeded (yes, I am delighted they succeeded after a week of failures).

I'm watching very closely and keeping a spreadsheet of what's happening to share in the JIRAs and with the Commerce team. Maybe they can figure out the commonality.