software project complexity

You are always wrong about the complexity of the project.

You overestimate it, and have been from the beginning. I'm not speaking of the resources needed for implementation, but rather about complexity for understanding (and almost always more resources as a consequence).

It's only that you don't know or have the right tools and you have much more complex understanding of the project. Your design is overengineered. You build a framework for representing graphs etc when a 20-line algorithm would do. Some of the tasks you are solving are no tasks actually with the right toolbox.

If you knew its actual complexity, you'd laugh at how much time and effort is spent at the project, and also how many people work on it. You think — what? Due to some lack of time the way you do is maybe 20% more complex than it would be if only you had time for analysis? Bullshit. It's 10 or 100 times more complex. The problem is other people have an even worse idea.

Following K.I.S.S might be right: there are less variants among simple implementations, and yours is one of them.

Thoughts

Today I've found a sheet of paper where I've written some of my conlusions several years ago after implementing a relatively large subsystem of the system I was working on in a large company.

1. To learn you have to make errors. You have to praise error rather than be afraid of it. "If you speak less and less, you'll have to speak more and more" — Cortasar wrote in his "Exam" meaning that not speaking makes the language and your command of it and expressiveness theirof degrade, and losing its expressiveness it makes you build larger texts. So, the same applies to errors: if you do less of them, you'll have to do more later.
2. You shouldn't be too clever in trying to overcome a difficulty: a much more efficient way is trying to avoid it altogether. So, you need to think about the nature, the source of the problem. Perhaps *it* can be removed. Being too clever in overcoming problems, seeing at once a (long) path of fixing a problem can obscure a (short) path that bypasses it altogether.
3. Relying on data format (think XML) in many different places weakens ability to change (and to support): changes in the format will lead to changing all places of using.
4. Premature optimization is the root of all evil (see (2)).
5. Data that differ slightly (say, two similar formats) are dangerous. You have to repeat yourself or at least are very urged to (things differ). Alternatively you can work out rules of conversion, but that conversion will most probably break when some change is introduced. Say, we have one source of data and all just works, we don't even think that there's some part of the code that builds another format upon it. We change the source format substantially and things suddenly break.

Corollary

Oh, and one simple thing my ruminations lead to is: the prototype must be written by the same people who will write the actual application, or else there's no point in it: the actual developers won't learn anything from it.

Solution by coincidence

Pragmatics have coined a term "programming by coincidence". Generally it describes a way of coding when a developer tries something without inner understanding of why it works, it works, it's considered to be the final solution, the scheme repeats.

It's interesting to look a bit ahead at the moment when the product has grown substantially and is used by the customer: what does the customer have at this moment? And I'm not only speaking about the situation when coding was precisely "coincidental" — it may have been a series of steps slightly compromising design for features with full awareness. It may have been ignoring 10% of requests that  will eventually lead to substantial changes in the product. It may have been failure to refactor and redesign along with new requests. It may have been lack of communication with the customer, or customer's drastically altered plans or improper architecture or platform choice.

So, what is it that the customer has in hands? From the purists standpoint, it's probably but a big ball of mud: hard to understand, hard to change, next to impossible to evolve.

On the other hand, it does the work for the customer. It eases his life (or life of his clients) and automates things. Re-doing it incurs additional costs, and some claim that it should by no means be rewritten from scratch, if that is your first intention:

"The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they've been fixed. [...]
Each of these bugs took weeks of real-world usage before they were found. The programmer might have spent a couple of days reproducing the bug in the lab and fixing it. If it's like a lot of bugs, the fix might be one line of code, or it might even be a couple of characters, but a lot of work and time went into those two characters.
When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work.
You are throwing away your market leadership. You are giving a gift of two or three years to your competitors, and believe me, that is a long time in software years."

Then, each of the requests successfully addressed is "documented" in the code, by the code (of course there are no tests in our muddy product!). The product can even show this specification of things to the person operating it.

So it's definitely worth something, but at the same time there's something pathetic about it... I think the situation should be cleanly stated by the development team to the management so that they planned for some "curing" phase. And the management — they should probably have some objective metrics that will show them — unless they can "feel" it — that this kind of situation has arisen.

Make it run, make it right

The practice of first making things run and then proceeding to making them right in software development is something I've always been adhering to — at least for each particular "atomic" task. Advantages are numerous: you get a proof that something is possible and/or understanding how hard technically that is, you and your customer (or analyst or PM or whoever is interested) can see the result and change things until it's too late, and the risk of not doing the task on time is reduced.

However, it still leaves a lot of room for choice for the developer: suppose you have an iteration during which you are planning to do 10 features. What will your best strategy be: trying to make each feature in its entirety by first making it run and then refactoring it to the point where you consider it to be right? Or rather make all 10 run first then refactor? Choosing somewhere between?

Each of the strategies has its good parts and its downsides. If you are doing everything run, the customer may ponder a feature, get some UX on it and come back with a change later until you started refactoring it. You also start refactoring with more requirements in mind and can make more accurate decisions. On the down side, if no time is left for refactoring you are left with a pile of hacks; and while making things run this piling up of hacks can also slow you down substantially: you don't have right abstractions. Interaction with other developers is also substantially hindered as you are not communicating your intentions clearly to them. And there's also a risk of the customer starting to seek perfection and drown you in changes until you get a chance to refactor.

Downsides of this approach are clearly bright sides of the contrary one and vice versa. If you don't get your features done during the iteration, they are likely to be moved to the next one. Release date may be shifted further and further and the product is put on risk.

So as the bottom line I'd say this: if your customer wants to have something that can be used, promoted etc. while the next version is being developed, be it with some warts and errors and things not perfect, and if both you and the customer understand that the first version will not be supported and is likely to be more thrown away than refactored, and if the whole thing is not so monstrous that it will start falling apart until you are finished (either because things get tangled for yourself even or for lack of communication in the team of developers) then you are probably better off with the "make ALL run, then make it right".

Parameter objects for API

I've always been interested in which paradigm to prefer:

declare functionName(concreteArg1, concreteArg2, ...)

or

declare functionName(containerArg)

Here, containerArg is an argument of our type with methods returning concreteArg1, concreteArg2 etc.

I usually preferred the former way as it allowed more concise implementation body and also the method signature speaks for itself. And also we don't introduce a new type. And the calling code is shorter.

There's a substantial difference in favor of the latter though: if later you decide you need to pass some additional parameters (or you don't need some), you can easily do that if you are passing an instance of your own type, but you are screwed if everything is specified in the API: you cannot but break API (or add another almost duplicate method), and even if you are the only user of the API you need to change calls and implementations etc. in every single place.

Such a simple thought and has never crossed my mind until I stumbled upon it.