On the Virtual Worlds Podcast this week we mentioned the rolling restart that updated a part of the Linden Scripting Language, the code that makes much of Second Life work. The update removed a feature, or fixed a bug, depending on your perspective. Chaos broke out accross the grid.
Lear Cale was building a ferris wheel whose occupants started spinning in their seats. Homer Horwitz had a series of multi-pose chairs that now had avatars embedded in the arm rests. Nack Barnes had an entire product line that couldn't be released, and a bar that nobody could sit in anymore. Charltina Christensen was planning a giant event across two sims (a wedding) when suddenly every single bit of furniture broke. A massive amount of content was ruined, and businesses were in jeopardy.
I contacted Soft Linden, the technician responsible for resolving the issue, to find out more. They declined to comment, and I was referred to the company JIRA; a central database for bug fix and new feature requests. The bug fix request to put things back the way they were is still there, and tells quite the story. Officially closed now, it garnered 93 votes and over 50 comments from desperate business owners pleading Linden Lab to roll back the clock. Dellybean North wrote: "Please, PLEASE restore this! You've just torpedoed a 20000L product my partner and I released a couple of weeks ago!"
While Soft Linden gave frequent updates on the progress of this fix in the comments, and was hailed by all those I interviewed for their speedy and attentive service on the issue, there was some confusion over why this was done in the first place. We know that the adjustment was to prevent sim crashes, but many felt that the particular method used was unnecessarily abrupt. This could have been done in a way that didn't disrupt content, and some who caught wind of the change ahead of time tried to warn the responsible Linden to no avail.
Instead, they debated whether the technique of placing an avatar in a specific position using this function was a "hack" or a "feature". Both Soft and Qarl Linden (who later joined the debate, although what part he played in the issue remains uncertain) assert that the use of this function in this way was a hack, while others felt that any use of a completely undocumented function could be thought of that way.
Of course, some questioned whether or not the Lindens should really be debating any of this at all. "The problem here is, that Qarl and Soft actually have the nerve to argue if this was a hack or not after there are dozens of content creators complaining that this breaks about 80% of all high quality furniture at the moment," stated Juliet Ceres in her comment.
Ultimately the issue can be blamed on a lack of communication on the impact of planned changes. The consequences were lost profit for many business owners, and an uphill battle to restore consumer confidence. Ilana Debevec lost three days of business while the problem was being worked on, and Rifkin Habsburg's new game that had just launched saw a sudden halt in sales that is only now beginning to recover. Events like these reinforce that nothing is certain when working in a beta environment like Second Life. As Ilana put it in later interview: "You can take every precaution and test you want when devloping a product, but you can't predict when you're gonna get SLcrewed."
The basic fact is that almost every use of LSL counts as a "hack" in some way as far as the scripter is concerned, because who can possibly know what isn't a hack and what is? Documentation is limited to what functions take which types of parameters.
I, personally, always assumed that the llSetLinkPrimitiveParams technique here was entirely official. After all, every time an avatar sits on an object it augments the link set, the av is treated as another linked prim, and that has been the case for years now; it triggers the changed event with CHANGED_LINK and so on. Obviously there are certain functions that would not apply to an avatar in this context, like setting a texture, but changing position is a logical progression, and dealing with an av as a linked prim in a linkset clearly has precedent.
I have the utmost respect for Qarl (and while I do not know Soft, I am sure he or she is very skilled) but to suggest that this is some sort of "hack" with the implication that it is scripters' own fault if it is broken is insane. And annoying. When there is no documentation, everything is a hack, and it is well known in the scripting community that the real skill is knowing how functions _really_ work, as opposed to how they _should_.
The hack versus feature issue the Lindens got confused about shows the large gap in communication which exists both internally and between LL and the community.
Since the introduction of het-grid the manner in which the lindens apply updates to functionality has overtaken their ability to consult with the community (and often themselves) about the possible outcomes of these updates. The blog is too broad (having to find a happy medium between the general public and techies) and the jira too pointalistic.
There is a big gap in the middle where there appears to be no comprehensive information about what the lindens are about to do to the grid, with enough lead time for techies to respond in an open forum and inform them what it'll break.
There needs to be a split in the blog communications between a public service blog and a grid/tech blog. This would provide a way for new features or service interruptions to be announced to the general public, and for seperate discussion of these features and their implications to take place in a seperate and appropriate forum.
I think it is unfair to call SL a beta due to these technical issues, when even microsoft can break hundreds of thousands of installations with an ill considered rolling update. These things happen. The issue is predominantly one of communication. The debate about this update should have taken place before the update rather than after. Het-grid updates are a technically sound idea, and I applaud the Lindies for implementing them. Their process of rolling out updates though leaves something to be desired. In the words of Thinkerer Melville, "we are the beta."
@ Pavig Lok:
I would say that sldev mailing list + Office Hours may fill the communication gap you're talking about.
I haven't been following this issue too much, and it hasn't affected me, but it's really basic software engineering-- once users are using and relying on something, it's a feature, and you have to keep at least the option of that old behavior available. When a module or function's interface or important side effects change, it's fundamentally a new thing, and you need to reflect that by giving it a new name, documentation, etc.
as stated above theres no documentation or communication
docs are user made by testing and useage, so a linden could just wake up and decide that anything that tickles their fancy is wrong and it should be fixed
the communication is one way only, they tell you what they want and theres not much of any chance to give feedback, so when "fixing" issues they have absolutely no idea what is actually being used whether its a hack in their eyes or not (cause again the only info we get is what is working live on the grid)
also this is not the first time that this situation has happened, altho in the past it was just major annoyances, now since (i guess) they feel that they got away with those without any community issues (which there was but again they do not listen to us) that they progressed onto breaking content
it boils down to how they run their house, theres no real management, their "tao" is to do what you want to make SL better, what this results in is what i call "post it note features" dumb upgrades and fixes that a team can whip out in the shortest amount of time possible, just to look busy to the boss before they always fail or backlash in some dramatic way while leaving major bugs for sometimes years
now im not a professional software developer, but in my experience with homebrew if i release a product with no documentation, no way to communicate with the users, and a i dont give a hoot about you attitude, it would fail
if it would fail as a hobby i have to ask myself "is this a way to run a company? and do i really want to invest my time and effort into this toy?"
I think they story here is that this got fixed, and rather quickly -- not that it was broken in the first place.
This wasn't the first time Linden Lab carelessly broke content, and it won't be the last. There used to be a game Primmies that was broken in exactly the same way (not llSetLinkPrimitiveParams, but with an update that broke its scripts). That game's owner decided it wasn't worth the trouble to patch his game, and it died.
Every content creator must realize. Developing content in Second Life is a constant battle. You have to expect things to break every so often.
Seems to me that SL imagineers need to read The Mythical Man-Month, which covered the nature and necessity of working with undocumented features/bugs in any system, and did so during the mainframe era. No wonder SL is dying.
@ Ugga Wugga:
I didn't notice that Second Life was dying. I'm not even sure this is actually happening. Let's talk about it again in 6 / 12 / 24 months.