Scrum is actually part of the problem, IMO. I've seen many teams turn scrum into a hammer and treat all future problems as nails.

Example problem: The foobar story has failed failed for the third sprint in a row.

Likely discussed in retrospective (plausibly good ideas, mind you):

- We need to break down stories more before we estimate them.

- Or we need to stop underestimating foobar stories.

- Or we need to focus on unblocking subtasks related to foobar stories.

Probably unconsidered:

- The foobar code is a mess and needs to be refactored.

- Or the foobar subsystem is too coupled to the Fizzbuzz subsystem.

- Or the need for some developer tools to increase productivity in the foobar ecosystem.

Since scrum is methodology oriented, methodology is the first tool teams reach for when a problem is encountered. And I see this after team leads make it explicitly OK to discuss technical subjects in retrospectives.

I'm not a psychologist, so I can't describe why this phenomenon happens, but I see it regularly.

Pretty much every kind of deadline driven development ramps up technical debt. Scrum certainly isn't the worst in this respect (developers make their own deadlines, and conscientious ones will build the time in), but the emphasis on commitment and the pressure to deliver at the end of the sprint puts pressure on developers to cut corners.

The worst part though, is that the product owner is usually non-technical and will deprioritize stories to clean up technical debt as a result.

IMO for any kind of development methodology to work it must have an opinion on technical debt. Scrum doesn't.

One of the few defining characteristics of scrum is that the developers define how much they can achieve, and this estimation is improved over time. If this is not happening there is something else wrong with the culture and Scrum is being used as a scapegoat.

* The prediction is made in a meeting while your head is "out of the code".

* The prediction is made in a group setting, rendering the decisions more easily subject to peer pressure and groupthink.

* The prediction is made up to 2/4 weeks in advance of actually doing the work.

* The prediction is made without risk of overshoot attached. Risk is critical metric which scrum conceals.

And the main defining characteristic of scrum that leads to pressure, after all of that unwarranted optimism:

* The prediction is designated as a commitment.

* while he is actually writing the code (so not up front)

* not in a group setting but as an individual, so either one person estimating the whole thing or each person giving different estimates

* (third point same as first, dont want to estimate up front)

* must incorporate what is often called 'contingency' (which is actually what the whole point of measuring velocity is for!)

* and the final point - he doesn't want to have to commit to it

how can you _not_ read this into it?

How is that the same as not being "required to give any estimate at all"?

> he doesn't want to have to commit to it

why not? an estimate is an estimate, not a commitment. Committing to an estimate makes it a commitment, not an estimate.

I might expect a dice roll to be 3.5, I'm not committing to the next roll being 3.5 - analysis should inform policy, in this case expectations informing stated commitments, but the two are not the same.

Furthermore, this bullet point actually takes the quote out of context - He specifically doesn't want to commit to the estimate produced under the previous conditions, not that he won't commit to any estimate. The difference is choosing to commit to an estimate you have high confidence in, versus any estimate given automatically being a commitment (where estimates may be required on demand).

Scrum people believe that scrum is the simplest way of measuring that. But at some stage you have to estimate the constituent parts of the project in order to get an idea of its size, and for those estimates to be useful in tracking your progress you have to do it in advance.

I repeat however, if you dont need to do this then thats fantastic! Many of us do however, and some of us choose to use scrum to do that, and some of us have had a great deal of success with that.

(edit: I worry that this sounds condescending. I am just trying to keep the tone friendly)

In advance of what? The only constraint on a useful estimate is that is comes before the task is finished - it needn't be considered as credible at the earliest possible time.

Also, your response doesn't really address my post..

I am clearly not expressing myself well. I am talking about a situation where some stakeholders are expecting a complete picture of roughly how large the project is and would like to be able to track how far your team is through this project on a regular basis.

I am putting scrum forward as a methodology for, in as short a time as possible, measuring the size of that project in a meaningful way by merely breaking it up into as small pieces as possible and attaching numbers to those pieces, intended to measure the size of each piece relative to the other pieces, and then over time discovering how long it takes to complete a piece of a given size.

> Assume each person giving different estimates for their own work, but not up front - ongoing as code is written.

The situation I outlined above (the time when scrum helps out) requires you have a stab at estimating all the constituent parts of the project at the beginning of the project.

> an estimate is an estimate, not a commitment. Committing to an estimate makes it a commitment, not an estimate.

True, but the point of estimating in scrum is to assign relative sizes to the pieces of work, not a number of hours, so this isnt a commitment to finish at a specific time but just to say 'I think this is one of the larger pieces of work in this project.' The person I was replying to sounds like they are on a bad team/project where people use their estimates to blame/finger point, and they are ascribing this to scrum as if the team wouldnt be doing this otherwise.

And in case you suggest that estimating without ascribing a time value is not meaningful, it is used to track how far you are through the project, and over time you refine what the finishing date will be given the emerging velocity.

> I might expect a dice roll to be 3.5, I'm not committing to the next roll being 3.5 - analysis should inform policy, in this case expectations informing stated commitments, but the two are not the same.

The analysis comes in discovering the velocity. The expectations evolve over time. But knowing your velocity is of limited use if you dont have an estimate of the overall size of the project.

> The difference is choosing to commit to an estimate you have high confidence in

This is the method for getting confidence in your estimate. You have an overall number of 'points' in the project and you learn how many points you can tackle on average every X weeks.

Every time you try and infer what I'm "really" saying or what "really" happened to me you get it completely wrong. Next time you do that just assume that you're wrong, it'll save us both time.

The blame/finger pointing on my projects wasn't really external (although in a different environment it certainly could have been). Developers themselves felt bad about missing their 'commitments'. The pressure/blame was largely self-inflicted.

Despite feeling bad the predictions were still consistently optimistic and still consistently wrong due to the environment the predictions were made in. It was a bug in the scrum process that led this to happen, but the team and management (and you, apparently) would rather assign blame to anything else other than a bug in their methodology.

>The analysis comes in discovering the velocity.

Velocity isn't a useful metric.

>This is the method for getting confidence in your estimate.

Except it doesn't work. It didn't work for us and it probably doesn't work for anybody else.

Confidence in estimates means treating risk and uncertainty as if it is real rather than sweeping it under the carpet, like it is in scrum.

Confidence means a prediction process that doesn't make developers feel guilty about being wrong, like it does with scrum 'commitments'.

Confidence a prediction process that doesn't intentionally subject developers to groupthink and peer pressure by immediately putting them on the spot like scrum planning pt 2 does.

Confidence means that your estimation process itself should be mutable. Under scrum it is fixed and not subject to review (if you change it you're doing "Scrum-but" and that's a sin, according to scrum trainers).

Most of all, confidence means that you should be able to inject technical debt cleanup stories into the sprint that derisk future changes. Scrum says that's only allowed if the PO says it's allowed. The PO is not responsible for missed commitments though, so it's not their problem.

Yes. I can take time out to answer email. I can take time out to make estimates as soon as I get an estimate request. Doesn't have to be done in a meeting.

>(so not up front)

What the fuck is the point of an estimate that's not made in advance???

>not in a group setting but as an individual, so either one person estimating the whole thing or each person giving different estimates

The latter. Is that a problem?

>(third point same as first, dont want to estimate up front)

"Not up front" is not the same thing as "not 4 weeks in advance". I'd do it as soon as the PM needed it to do prioritization.

>must incorporate what is often called 'contingency'

If you think risk and contingency are the same thing you're an idiot. Risk is story A (e.g. upgrading dependencies) might take 0 hours or might take 4 weeks while story B (updating translations) is going to take 1.5 hours and it's really only going to take 1.5 hours.

Contingency is (for example) "let's make sure we have 4 weeks spare before doing story A".

>(which is actually what the whole point of measuring velocity is for!)

No, velocity is about measuring how fast you're doing stories.

>and the final point - he doesn't want to have to commit to it

Yeah, because as soon as you start assigning blame for missing feature deadlines the technical debt dial gets ramped up to 11 and predictions become an exercise not in being accurate but in CYA.

An estimate about how long something is going to take can be wrong for many reasons that aren't the developers fault - bugs in libraries, technical debt in dependencies, technical debt they weren't aware of and didn't create, team members disappearing, etc.

If you want developers to commit to things make sure it's things that they have full control over.

I am here assuming that you want to be able to try to measure your progress through the project (as I mentioned, this is the only thing scrum does for you). Both of you seem to be suggesting (dont insult me if Im wrong) that this isnt the highest priority.

And no, velocity is to make the whole system self-adjusting. If I put 3 points against a story we use velocity to discover over time how long those 3 points take. This self-adjusts to incorporate for contingency.

If you disagree with this then we simply disagree on what velocity is about. It doesnt make us enemies, we dont need to get super pissed off at each other.

Scrum is complex and not always possible to follow exactly, so this is to be expected but it makes me wonder, how many successful projects are out there that are following the true Scrum methodology?

My guess is that it's a few more than the classic waterfall but I still seem to see far more failure than success stories.

Regarding success stories, it might be that process doesn't play such a critical role as long as solid engineering techniques are used and the team is competent.

All those methodologies are for the less stellar programming teams, to get consistent results from those (also to a lesser degree to make good and bad programmers work well along each other). Because you can't always get the best programmers.

If Scrum would only work well with good programmers, it would be next to useless.

It remains to be seen if big Scrum engineering projects where Scrum is actually applied even exist. I can't even think about one on the top of my head. I'm not even sure Scrum is that well defined for us to be able to judge if is correctly applied or not. And it's yet another story to judge if they are successful or not.

In the end it does not matter much. The theoretical vision that nobody ever uses has almost no interest if you are concerned with real world efficiencies.

You are engaging in equivocation.

> Want to construct a bridge or a rocket, design a microprocessor? You are not going to do that with "stories".

Nor are you going to use the software development methodology described as the waterfall method (you may use a physical engineering methodology that was among the inspirations for that software development methodology, but those are distinctly different things, with different specific practices, and different domains.)

> I'm not even sure Scrum is that well defined for us to be able to judge if is correctly applied or not.

Scrum is exquisitely well-defined, both as to what it involves, what it specifically excludes, and what it is neutral to, in the Scrum Guide. (There's lots of confusion between Agile, a broad approach which is not a specific methodology, and Scrum, a very-specifically-defined -- though by itself fairly incomplete, in that any implementation of Scrum needs lots of decisions on the things to which Scrum is neutral -- methodology.)

Scrum origin is partly in manufacturing. Now there are some common points between some aspects of software dev and manufacturing, especially more so if the software being developed can be iterated very quickly (but very few if it's not the case), but at least in the real world (and maybe even in the theory) Scrum is what is also actually mainly used to interact with other stakeholders. And given how the communication is performed, and its content, that might be better than complete chaos when nobody is actually able to do the work they are supposed to do (PM being limited to having vague ideas, lack of a truly competent tech lead doing actual tech lead work, lack of vision by management, and so over) and only very vague general ideas of what should do the software -- or more generally the whole product -- are ever emitted.

As soon as "serious" stuff starts to be involved, you need real boring engineering, with functional analysis, requirement engineering, modeling, systematic testing or even partial proofs, etc. And you need to have it structuring communication between teams, and day to day work. And then, I don't expect Scrum or anything Agile in such a context adding any kind of value.

Now the theory of Agile and Scrum has evolved because of criticisms to a point where we are told that it actually do not cover the things that matter. That is bullshit retro-justification, now that the world is fucked up trying to make sense how to use that. Here is the Agile manifesto:

> We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

> Individuals and interactions over processes and tools

> Working software over comprehensive documentation

> Customer collaboration over contract negotiation

> Responding to change over following a plan

> That is, while there is value in the items on the right, we value the items on the left more.

Engineering is mainly about "processes and tools", of course "individuals and interactions" are also needed, but there is no need to oppose them (although I am not sure what is the point of "individuals" here; the authors might as well said "oh and by the way be nice")

"comprehensive documentation" is critical in all kind of domains, and now that software is everywhere it just makes no sense to declare your "preference" of "working software" above "comprehensive documentation". It is, again, even dangerous to oppose them.

Customer collaboration over contract negotiation; again, highly dependent on the field and specific project if this is something where it makes sense to even have a "preference" or not.

"Following a plan" is what you do about how you organize your work when you use Scrum. There is no problem in studying the impact of a change any time if proper engineering practices are used. Obviously, the cost can vary depending on various factors.

My conclusion about Agile and Scrum, is that if you prefer all of that (4 Agile preferences, and the Scrum theater), you should seek projects that are suitable for the Agile preferences, and so poorly defined that Scrum is a plus. On my side, I'm just not seeking to work on chaotic projects -- on the contrary I try to bring logical and more systematic practice where I feel that chaos reigns -- and I'm neutral about Agile preferences, I prefer to choose projects on other criteria (mostly; intrinsical interest)

And Agile does not avoid processes and tools, it recognizes that process and tools must be specifically fit to the particular team and context of work (Scrum, particularly, is a baseline set of processes and tools that is designed to serve as a framework for common contexts of software work -- its intentionally incomplete to avoid specifying too much that would narrow its scope of applicability.)

> "individuals and interactions" are also needed, but there is no need to oppose them

The need to oppose them comes from the authors' concrete experiences in the software world before writing the manifesto, where very frequently canned (often consultant-pushed) processes and tools were being adopted by management in shops without considering the dynamics of the existing team and the particular work being done. (One of the sad ironies of the Agile movement is that the "Agile" banner itself has become a tool for the same kind of thing.)

> "comprehensive documentation" is critical in all kind of domains

Yes, it is; the preference stated in the manifesto is, again, the result of concrete experience where projects were quite often focused on producing mandated documentary artifacts because there was a checklist and that was how "control" was exercised, but the documents required and delivered were often irrelevant to (and not consumed by, or updated to reflect changes resulting from, the process of) delivering working software.

> Customer collaboration over contract negotiation; again, highly dependent on the field and specific project if this is something where it makes sense to even have a "preference" or not.

This is intended specifically in the content of developing specific software requirements (and, really, its more about the dev team pushing the customer to engage rather than provide hands-off requirements.)

The Agile Manifesto really deals with concrete problems encountered in particularly enterprise software contracting (but bad practices from the enterprise world were, at the time, getting exported to the rest of software development, so not limited to the enterprise world.)

> "Following a plan" is what you do about how you organize your work when you use Scrum.

Scrum, like most methodologies that attempt to implement agile values, focuses quite a lot on managing potential rapid change within the plan.

Scrum is what you do when you try to do software engineering without actually doing software engineering. It insanely meta, and like explained in other comments, the improvements you get from its loop are too often meta (we should evaluate more accurately). I prefer to stick to the real thing, and core engineering practices. Scrum attempts to fix situation when core engineering practices are misunderstood and used as constraints instead of being used as something essential to the dev of a good product; but it is vain to try to fix such a situation by engaging key people even less in core engineering practices, and more in mundane discussions where the real problems are never addressed.

Oh, yeah, that's definitely a problem. I don't think the Agile Manifesto is bad at all, but I think that, ironically, in application it suffers from the same problem it sought to address -- people are looking for simple answers that can be applied without deep knowledge of context. The Agile Manifesto and Agile software movement was itself a strong reaction against that, but unfortunately it (and tools from within that movement, like Scrum) get applied by exactly the same process that the Manifesto was a reaction against (focusing on particular ways it had manifested, prior to the Manifesto, in software development.)

> Honestly if some management is stupid enough to force badly suited processes and tools instead of letting (competent) teams choose better ones, I doubt they will suddenly see the light by reading the Agile manifesto.

Absolutely; the real audience of the Agile Manifesto is software development practitioners that have influence with management, and its not really "new knowledge" as a concrete distillation of experience. The fundamental problem, I think, with Agile isn't that its ideas are bad, its that the real problem it deals with isn't a problem of process/tools, or even the meta-level approach to processes and tools, but a problem with institutional organization and leadership of large entities that happen to be doing software projects, and how that manifests in software projects.

The agile movement has produced some new tools that can be applied effectively in, largely, the areas that didn't really have the worst cases of the problems that motivated the movement -- because its helped motivate and inspire a lot of efforts by people with decent engineering backgrounds at finding new ways of working.

But the kinds of organizations that were worst afflicted by the problems that the Manifesto set out to address are still the most afflicted by those problems, and what they've gotten out of it is a lot of new processes and tools that consultants will sell them, their management will blindly adopt without understanding the conditions which makes them useful, and thus they find all kinds of new ways to fail.

> Scrum is what you do when you try to do software engineering without actually doing software engineering.

Scrum is largely orthogonal to software engineering (presumably, people using scrum in a software project will be doing software engineering within Scrum, but Scrum is not about software engineering.)

> It insanely meta, and like explained in other comments, the improvements you get from its loop are too often meta (we should evaluate more accurately).

Scrum is designed to be very meta, true. And, yes, if you mistake Scrum for a complete process rather than a process framework, you aren't going to get much out of it beyond omphaloskepsis. (I'm actually not convinced that Scrum is particularly valuable, even as a framework, as anything more than a well-known starting point to develop an appropriate, context-specific work model.)

Regarding development: My main point is that Scrum leans towards agile methods such as XP (testing, CI etc), but it also sucks the time necessary to do those things well. The time Scrum takes off of the devs' working hours could much better be spent on those.

There's slightly more to it than that: it also encodes an assumption that you're working with a single fairly-tightly-integrated group (with synchronisation points at least daily). It's possible that this helps with estimation and scheduling -- it's a lot less clear that it helps get the best outcome in other respects.

There's also something of an uncertainty principle here -- more precision of estimation is possible, at the expense of increased expected timescales (partly due to padding, partly due to picking lower-risk approaches).

I personally think estimating projects is one of the most difficult things about this industry. Especially if we're talking about delivering many calendar-months worth of effort for a team , unless its just a variant on some other project[s] the team is well experienced at

More like wandering in the desert, hoping you find the promised land.

Been thru scrum master training 3 times, been on many "agile" teams. I've never heard this rationalization. Rather, a common justification for "agile" was you always have a working product. Which might be nice if things worked out that way.

Also, PMI style critical path worked just fine for figuring out that "straight line".

Scrum and "agile" democratized project management, empowering every poseur to claim expertise and ability. Whereas PMI required real effort to learn and master, Scrum flavored "self help" books can be flipped thru before you finish your coffee and then safely stored in plain sight on a book shelf, never to be touched again, allowing said poseur to claim the daily mutant chaotic dysfunctional mismanagement that they've always done is now "agile".

If you are objecting to defining the scope as small tasks and measuring your progress through that over time, then continually re-evaluating this scope as requirements change, then I think you are not working in an environment that would benefit from this kind of tool.

Its just a pragmatic set of guidelines, and objecting to it with such ridiculous vitriol makes you sound as foolish as the people I think you're objecting to.

"with such ridiculous vitriol"

Emperor, little boy, no clothes. It's thankless work.

In opposition, defenders of Scrum et al use the No True Scotsman's fallacy. Because those of us who have tried and failed are just morons.

Project management is risk mitigation. In my experience, most "agile" projects have been risk amplifiers. Ironic.